This particular feature is something that's happening as a result of your using MacOS (i.e. it's not something that Sublime does natively; the OS is doing it on your behalf).

In the general case most MacOS applications should have native menu items to combine windows together like this at the user's request in combination with the Prefer tabs when opening documents setting in the Dock area of the system preferences.

Sublime doesn't support the native menu items for this (yet), so it relies solely on the setting; having it set to Always (and also In Full Screen Only, but this tends to cause problems with Sublime) makes MacOS automatically "tab" new windows. Setting that setting to Manually stops this from happening.

This is also somewhat controlled by the Theme that you're using in Sublime. For MacOS, a Sublime theme can theme the menu bar of the window to match the overall application theme. Behind the scenes, this makes Sublime declare to MacOS that it wants to be in charge of displaying it's own window, which stops the system from automatically combining windows together.

An example of a theme that does this is the Adaptive theme that ships with Sublime.

The reason it doesn't work is that you have the wrong alias. Here's the correct one for Sublime Text 3:

alias subl='/Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl'

This is an alias that Sublime Text specifically provides for macOS. It will not block execution and—by default—open files in a tab next to the ones that are already open.

If you want to know more about why your alias didn't work, read on.

Application packages in OS X have an .app suffix, but when launched, they actually launch a binary that's specified in Info.plist in said package. This is the binary your alias links to, in the MacOS folder.

Running an application from the GUI or through Launch Services (e.g. via osascript or open) makes OS X open that binary, but at the same time ensure it's only launching one instance of that app. When you call the binary directly, you bypass this restriction and launch another instance of the application—unless the application has measures for preventing two instances from being launched, which normally OS X handles.

Also, as a side effect, launching the binary will block the terminal execution until you exit the program or suspend it to the background. Using the built-in subl from Sublime Text returns control to your terminal, as it actually uses a Launch Services call to open the Sublime package instead of addressing the binary in MacOS only.

Since you're basically launching a second Sublime Text instance, it'll show you the files it previously had opened, because that's its default behavior.