In the first place, I think it may be reasonable to ask why anyone should be confined to 4GB storage in 2016. Most smartphones, even the very cheap ones, have at least the double of that. But, on the other hand, this can be done.

1. As has been suggested, start from Debian live, and modify that by adding packages, using persistence much the same way as in Knoppix. Simplest and safest, but it might not get you where you wanted.
2. Knoppify Debian live. There is a recent thread about this here.
3. Purge and remaster. It is possible, not even very hard, to slim DVD Knoppix down quite a bit. I routinely purged a couple of GBs worth from stock Knoppix before starting to add packages, so I guess the basic image would be around 3.3 GB. I'll detail this a bit here.

Do test runs without persistent store mounted (knoppix noimage). You might not believe it, but the Debian package system seems to be very buggy when it comes to package removal. I believed this was fixed, but it isn't. So when you remove a package, it has the tendency to remove everything that package is dependent on, even if other packages depend on it, making a potentially huge cascade. Maybe this is due to circular dependencies being introduced (which should not have been allowed in the first place), I don't know. But the effect is that if you try to remove something as peripherical (to the average user, not to me) as TeX, you end up having things like LibreOffice, Okular and Pidgin removed. (Aptitude is even so rude as to complain about it, but say "as you requested" when it performs some of these faulty removals. I guess there are quite a few infallible popes around.)

Removing games seems to be safe (freeing about 750MB uncompressed)
Code:
sudo aptitude -y purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^games | cut -d' ' -f2`
sudo aptitude -y purge `deborphan`
But this one upset Knoppix completely
Code:
sudo aptitude -y purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep ^tex | cut -d' ' -f2`
After the two purge command, about 500 packages were removed (down from 3872, ie. average compressed size on the DVD about 1M)
It should be possible to fix the situation afterwards also in this case, though. Starting with something less essential (to Debian it seems) than TeX might be safer.
Also removing whole sections, like it is done here, is very dangerous compared to removing on a package basis

For example, removing inkscape should free at least about 100MB
Code:
sudo aptitude -y purge `dpkg-query -W --showformat='${Section} ${Package}\n' | grep inkscape | cut -d' ' -f2`
sudo aptitude -y purge `deborphan`
Omitting the -y switch with aptitude gives you more of a negotiating mode of operation. So the catastrophes may be stopped.