Actually it seems that you might pass to each new connection in connect in your user_socket this particular PID or name of the Agent and it should solve this case. You only need one Agent, so you can name it, eg.

Agent.start_link(your_func, name: :myagent)

and later get from it

Agent.get(:myagent, your_get_func)

Also you can consider to use Registry, which is from Elixir 1.4 in standard library.

This agent can be in lib and you can just add it in the <app_name>.ex file as a single worker and supervise it with dedicated Supervisor if needed.