Whether you have downloaded the binary DLLs or compiled from source (doc TODO) you will have 3 or 4 DLL (depending on whether you have build XNua runtime for both x86 and X360).
Copy these to a directory somewhere (its largely up to you) then open up GSE with your game project. Its important that the PC version of XNua.dll, XNuaCompiler.dll and XNuaContentImporter.dll all live in the same directory. This allows the assembly loader to find the types without installing in the GAC (I suspect you could do this I guess, but honestly haven't a clue :-P)
Right click on the project properties and click on Content Pipeline, click Add and add the XNuaContentImporter.dll.
If done correctly it will look something like this 
Now add a reference to XNua.dll to the project (pick the correct version for which ever target this project is for) and you should now be able to add .lua files to your project and have to compiled into .lil files.