From your post it sounds like the first person in the office will open the database by file sharing to the server and opening it. This person then becomes the host to that file - not the server.
Then when others try to do the same, it will tell you that the file is in use - which is correct.
The above is an assumption, so I should ask if the Filemaker databases are firstly opened on the server before anyone acceses them. If not then you will not be using the server as a server, merely as a place to store your files
As you are not using FM server your files will need to be opened and closed manually. There is also a question of how you back up your files in this scenario - I have set up a few systems such as this, and getting people to remember to close down the files before they back them up is always a problem - sepecially if they are on a remote machine.
FM server would be your best solution, as your files can be open 24/7 and you will not come across your error ever again.
If you only acquire FM server for the automatic back up facilities, then it is well worth the expense. I have fitted many backup systems and the question I always put to those who think that money is better spent elsewhere is, how much would it cost to retype in the data- on the assumption that you still have the data to re-enter.
Consider looking at FM server as a solution, it may seem like a large outlay initially, but the time saved on Admin and backup will pay for itself in no time.