As someone who designs computer systems, I can agree with the suggestions so far. Certainly, the operating system itself is an example of systems software. However, there are several others that have not been mentioned so far. From my perspective, the system software is the code that is required to enable the application programs to execute. In many O/S variants, there is a distinction between the user-state and supervisor-state within the processor. Things that live in the user-state are applications, libraries, and engines needed to run other user software. The supervisor state contains the main O/S kernel, device drivers, interrupt service routines (ISRs), exception handlers, the scheduler, kernel threads, and the like.
If we think about and O/S like Windows, then we see the Win32/64 or WinRT kernel itself (plus device drivers, ISRs, etc.). And a file system that contains user-state applications, DLLs, etc. There’s a similar distinction in an O/S like Linux. But, the file system in Linux is largely GNU-based (Glibc, etc.) for Linux whereas there is a completely different user-space for Android, Tizen, WebOS, etc. in spite of these O/Ses using the Linux kernel.
However, we have to also add the distinction that certain user-space software is an enabler and not an end in itself. For example, the C/C++ runtime, JavaVM, PythonVM, Glibc, socket libraries, etc. are not applications, but rather application enablers. Hence, in my mind as someone who helps develop these sorts of code, they also fall into the “system software” bucket because user applications could not run without them and yet the user application developer doesn’t have to think about them — they’re just there in the file system.
So, while I wouldn’t consider Vi and Emacs to be system software (they’re using applications that we use as editors), the compiler, linker, loader, etc. could easily be considered system software. Additionally, the libraries/DLLs would be considered system software because they are a resource that is developed in a very particular way to aid application programmers to develop and run their code.