Last Update: Tue, Jan 2, 2007 (1330 Hours St. Louis Time)
$n1p2> ssh -g -L 8080:n1p3:80 n1p3"Connect to host n1p3 port 22: Network is unreachable." What could be the problem?
onlusr> source /users/onl/.topology.cshI get an unexpected end of file.
http://localhost:8080/~YourUserName/syndemoto the browser I get Unable to connect and Firefox can't establish a connection to the server at localhost:8080.
(cd .www-docs/syndemo/Images; touchme &)the script did not run. The following error message was shown in the command-line window:
-bash: cd: .www-docs/syndemo/Images: No such file or directory -bash: touchme: command not found [1]+ Exit 127 touchme
sudo /usr/local/bin/sec/synsterthe following message appeared in the window:
"Sorry, user toshiba is not allowed to execute '/usr/local/bin/sec/synster' as root on onl41.arl.wustl.edu."where onl41.arl.wustl.edu is the external address of n1p1a (where the attacker resides).
Warning: Permanently added the RSA host key for IP address '10.0.1.3' to the list of known hosts.what does it mean and whats wrong?
A1: There can be many reasons, but it looks like you (and the rest of the students in your class) need to make your home directories world searchable.
I will change all of your home directory permissions to 711 (rwx --x --x) so you don't have to do this.
REPEAT: You do not have to do anything because I have already changed the permissions on JUST home directories.But if you want to know how to do it on your own and test it easily, read on. Do this:
chmod 711 ~UserNamewhere UserName is your username. This makes your home directory searchable by everyone including Apache.I tested this idea by changing the permissions on the home directory and then using a test tunnel that goes only over the control network directly to the host $n1p3 by doing this (replace UserName with your username and $n1p3 with the external interface name of the host on port 3):
- Make sure .www-docs directory is world readable/searchable
chmod 755 ~/.www-docs- Put a test web page in the .www-docs subdirectory
cp ~kenw/index.html ~/.www-docs/~kenw/index.html just displays a "Hello" message.- Build test tunnel:
ssh -L 8080:$n1p3:80 onl.arl.wustl.eduThis tunnel goes from your client to onl.usr to $n1p3 over the control network. It doesn't use the NSP router.- Enter http://localhost:8080/~UserName to your browser
You should see the message "Hello from ONL".
A2: It looks like you did not build your RLI tunnel. See the "Getting Started" link in the sidebar of the ONL web page.
$n1p2> ssh -g -L 8080:n1p3:80 n1p3"Connect to host n1p3 port 22: Network is unreachable." What could be the problem?
A3: I assume that you issued the following command from $n1p2 (where $n1p2 is either onl32, onl38, onl26 or onl11 depending on what NSP was assigned to you during the commit):
$n1p2> ssh -g -L 8080:n1p3:80 n1p3This means that you want to create a tunnel for port 8080 on $n1p2 going to port 80 on n1p3 and that the terminal session will log onto the host n1p3 through the NSP. Port 80 is the usual port that Apache listens for requests.The "Connect to host n1p3 port 22: Network is unreachable" typically means that either there is a hardware problem along the physical path to n1p3 or that there is no route to n1p3 or there is a problem along the return path from n1p3.
I suspect that the real problem is that you have no route established from n1p2 to n1p3; i.e., the route table at port 2 is not configured properly. So, now the question is exactly what is the problem?
- Step 1 (Find out if there is connectivity from n1p2 to n1p3)
Try to ping n1p3 from the host $n1p2 (you should replace $n1p2 with the correct interface name):onlusr> ssh $n1p2 $n1p2> ping n1p3If you get a normal response, the route is ok. If not, there is no route to n1p3. Assuming that you get no response ...- Step 2 (Look at the route table at port 2 of the NSP)
In the RLI, select on the NSP icon:Port 2 => Route TableYou should see default route entries:prefix/mask next hop 192.168.1.16/28 0 192.168.1.32/28 1 192.168.1.64/28 2 192.168.1.80/28 3 <== The important one ... etc ...The 4th entry is the important one. If the RT is empty, you should generate the default routes for all ports (if you are using the "syndemo.exp" config file, the default routes should already have been created ... let me know if you are using the file).
Create default routes for all ports by going to the RLI main menu and selecting:Topology => Generate Default RoutesNow, your ping command in Step 1 should succeed. So, repeat Step 1. If successful try making the tunnel again.- Step 3 (Try creating the gateway tunnel again)
$n1p2> ssh -g -L 8080:n1p3:80 n1p3
A4: Congratulations ... you have gotten very far.
Assuming that your ONL login name is mndhamod, I see that you have the file /users/mndhamod/.www-docs/syndemo/Images/touchme. Ahaaa. That script is incorrect because it refers to my home directory instead of yours.
Ok, I just modified it so that it should now work. All it does is touch the image files every 2 seconds and make them look like they have changed so that the browser sends a request to get the "new" image.
>>> Basheer <<< I put a new touchme script in
~onl/.www-docs/syndemo/Images/touchmethat should fix the problem that Mohammad was having. The rest of the class should replace their touchme script with that one.
onlusr> source /users/onl/.topology.cshI get an unexpected end of file.
A5: This looks like you are trying to source a c-shell script when you are actually running the bash shell. Yes, when I enter:
ls -al ~secI see files in your home directory like .bashrc. And when I enter:ypcat passwd | grep howaidiI see:sec:x:5261:5005:mohammad howaidi:/users/sec:/bin/bashwhich indicates (last field) that your shell is bash and not csh. So, you need to do this:onlusr> source /users/onl/.topologyi.e., source the file .topology, NOT .topology.csh.
http://localhost:8080/~YourUserName/syndemoto the browser I get Unable to connect and Firefox can't establish a connection to the server at localhost:8080.
A6: This last problem indicates that you have not properly setup the tunnel at your client host so that traffic to port 8080 will go to the relay node. In the instructions for Approach 1, it says that you need to build the A-B-C tunnels like this:
client> ssh -L 5050:$n1p1a:5050 -L 3552:$n1cp:3552 -L 8080:$n1p2:8080 onl.arl.wustl.eduwhere you need to replace $n1p1a, $n1cp and $n1p2 with the appropriate ONL host names which you can get by clicking on the n1p1a icon, the NSP icon, and the n1p2 icon (e.g., onl35).These instructions are about 3/4 of the way down the SYN Demo Web page.
(cd .www-docs/syndemo/Images; touchme &)the script did not run. The following error message was shown in the command-line window:
-bash: cd: .www-docs/syndemo/Images: No such file or directory -bash: touchme: command not found [1]+ Exit 127 touchme
A7: I am guessing that you ran it from some directory other than your home directory.
I looked in your home directory ~toshiba and you have the directories:.www-docs/ .www-docs/syndemo/ .www-docs/syndemo/Images/So, that looks ok. The command you ran needs to be run from your home directory or you need to change it so that the path .www-docs/syndemo/Images is correct. I just su'd to your account and tried the command and it worked as expected. So, try on your $n1p3 host:cd (cd .www-docs/syndemo/Images; ./touchme &)The first command changes to your home directory. The second one executes the touchme command. If you want to kill the touchme command:ps -l kill -9 Xwhere X is the PID of the touchme script.
sudo /usr/local/bin/sec/synsterthe following message appeared in the window:
"Sorry, user toshiba is not allowed to execute '/usr/local/bin/sec/synster' as root on onl41.arl.wustl.edu."where onl41.arl.wustl.edu is the external address of n1p1a (where the attacker resides).
A8: I pushed out a new sudo file to resolve this second problem. I tried it with your account on onl41, and it worked. So, try it now. The problem should go away.
A9: Right. It looks like the default command search PATH for most users does not contain the current directory ("."). I will have to update the Web page.
A10: [[ THIS IS THE IMPORTANT PARAGRAPH ]]
It looks like you told the RLI that your plugin directory was /users/s3curity/plugins. If so, it assumes that the directory will contain sub-directories with names of the form XXX-NNN where XXX is a plugin name and NNN is a plugin number; e.g., syn_demo-54203. Inside this subdirectory ~s3curity/plugins/syn_demo-54203/ should be the code to that plugin.I just looked at your directory /users/s3curity/plugins and it looks really messed up. But it now does have the subdirectory syn_demo-54203. So, you should no longer be getting that message ... even though the rest of the directory is messed up.
But if I were you I would just use the standard syn_demo plugin stuff supplied by default. Only if that plugin worked would I attempt to build my own version of the plugin. I think the config file in ~onl/export/Examples/syndemo/syndemo.exp uses the default plugin located at ~onl/stdPlugins/syn_demo-54203.
But if you really do want to create your own version of the plugin, and you are still having problems, I would recreate your plugin directory to conform to what I described above.
A11: It is very hard for me to debug this without knowing more information.
It sounds like you are saying that the browser cycling through a sequence of images (5?) and that you can control the image transmission by clicking the Stop/Start button in the Web Client. Right?
Q1: Does the Bandwidth monitor show the Attacker traffic?
I suspect yes. If not, that would be really strange.Q2: Did you deviate from the instructions in anyway? If so, how?
Q3: What does the 'netstat' command tell you about traffic coming into the $n1p3 host? For example, do this WHILE IMAGE TRAFFIC IS GOING TO YOUR BROWSER:
onlusr> ssh $n1p3 $n1p3> netstat -i ... You should see 3 lines (eth0, eth1, lo) ... ... Record the RX-OK and TX-OK numbers ... ... Wait about 10 sec ... $n1p3> netstat -i # again ... Record the RX-OK and TX-OK numbers again ...The difference between the second and first RX-OK numbers indicates how much traffic came into the host over that interface in 10 sec. The difference between the second and first TX-OK numbers indicates how much traffic went out of the host over that interface in 10 sec. The interface with the smallest numbers is probably the one going into the NSP. The other interface is connected to the control network.Unfortunately, the control network is sometimes on eth0 and sometimes eth1. So, I can't tell in advance which interface it would be ... unless I knew the external host name (e.g., onl31, onl37, onl25, or onl10). Now, in these 4 cases, I think eth0 is attached to an NSP and eth1 is attached to the control network.
The command "ls -l ~/.www-docs/syndemo/Images/*.jpg" indicates the sizes of the images range from 83735 bytes (evening.jpg) to 108161 bytes (nudy.jpg); i.e., about 100 KB per image. So, we expect that if image requests are sent every 3 sec, that in 10 seconds, you will see a difference in outgoing traffic (image) of about 300,000 bytes and a small amount of incoming traffic (HTTP request).
If I had to guess, the image traffic must be going over the control network instead of the private network (NSP). And that would only happen if Tunnel D were not built properly. This is only a guess. But if you did this:
$n1p2> ssh -g -L 8080:$n1p3:80 $n1p3instead of this CORRECT WAY:$n1p2> ssh -g -L 8080:n1p3:80 n1p3when building Tunnel D what you are seeing would happen. This last tunnel forces the traffic coming into port 8080 of the relay node to go to port 80 of n1p3 over the interface to the NSP.Another possibility is that you built Tunnel D ok, but for some strange reason, the route going from n1p2 to n1p3 was not built properly and somehow the relay host $n1p2 found a path over the control network to $n1p3 (unlikely, but possible).
Of course, if the above fails, we could verify all of this ... painfully ... using tcpdump on the right interfaces along the traffic path and deciphering the output.
Warning: Permanently added the RSA host key for IP address '10.0.1.3' to the list of known hosts.what does it mean and whats wrong?
A12: The short answer is that there is nothing wrong.
10.0.1.3 is the IP address of the eth0 interface to onlusr. You can see this by enterring:
onlusr> /sbin/ifconfigwhile logged into onl.usr and note the inet addr field in the eth0 entry. You didn't say so, but my guess is that you got this message when you tried to build an SSH tunnel from one ONL host to another. More specifically, you must have enterred the ssh command FROM onl.usr to some other onl host.Whenever you SSH to a remote host X from host Y, the IP address of host Y (the FROM host) is looked up in the file ~/.ssh/known_hosts (a plaintext file with RSA keys) at the remote host X. If it is there, then you are connected to that host. If not, then SSH will add the hostname to the file after authentication. In your case, I noticed that your ~tasneem/.ssh/known_hosts contains an entry for 10.0.1.3 as its first entry ... which makes sense. This is why ...
Your onl home directory is NFS mounted on every onl host which also means that the file ~tasneem/.ssh/known_hosts is accessible on every onl host. Suppose that you are on onl.usr, and you enter something like:
onlusr> ssh onl31All of your onl hosts (given to you through File => Commit) are setup to accept the ssh connection without asking for a password. But the ssh server (daemon) running on onl31 will still do some authentication. One thing it does is look at the file ~tasneem/.ssh/known_hosts on onl31 to see if the IP address of onlusr (10.0.1.3) is a host that you have allowed to login to onl31 before. The first time you do this, there is nothing in the known_hosts file. Since you are allowed to login to your onl hosts from other onl hosts, ssh adds the host to the known_hosts file.Enter the command "man ssh" and scroll down to the section "Server authentication" for more details.
A13: runTcpMon.pl produces the file tcp.data in your home directory. Your file contains all 0s. That script is looking at the file /proc/net/tcp. Enter the command "cat /proc/net/tcp" on any of the onl hosts and you will see that the file contains the state of TCP connections to that host. The script is just looking for connections to port 80 (HTTP) (or hex 0050) that are in state 03 (partial connection). I am guessing (and this is only a guess) that the runTcpMon.pl script is running on the wrong host. It should be running on your $n1p3 host.
A14: The fact that you got traffic for 30 seconds is good because it shows that things are working. The fact that it stops after 30 seconds will be difficult to debug. Furthermore, the demo is fragile. So, if anything goes wrong, it doesn't recover well. Some comments:
- I would first try it again and see if this behavior repeats.
- If the touchme script is still running, the problem is probably NOT due to the browser thinking that it already has up-to-date images. You can tell that the modification dates of the images are in fact being updated:
cd ~/.www-docs/syndemo/Images ls -l *.jpg ... wait 10 seconds ... ls -l *.jpg ... repeat occassionally to see that the modification dates are changing- I don't know if the RTT between you and us would be an issue. It must be about 150-200 milliseconds. It shouldn't be an issue since the traffic volume is low.
- I don't know if network security at your end would be a problem. Again, I don't think so since you are able to begin an experiment.
- I am guessing that either the Web server or the plugin got overwhelmed. I would try turning the attacker off when the images stop and see if that allows outstanding requests to drain. Then turn it back on. And repeat this process. But if you are having this problem when the attacker has never been turned on, that would be a different problem.
A15: I just tested this on your Web page and it worked. So, there is no problem with your files in ~mndhamod/syndemo/.
The fact that you got "Page not found" probably indicates that you are communicating with some HTTP server, but I suspect the wrong one. Probably some tunnel was not built properly. But it is hard to tell. You will have to find out where the traffic is going by working backwards from the HTTP server $n1p3.
For example, if you monitor the ingress and egress bandwidth at port 3 of your NSP, you should see some traffic going out of and coming into that port when you hit the carriage return on the URL. If your http request got to $n1p3 but the page was not found by Apache on $n1p3, you should see two spikes in the plot, one for egress (the request) and then one for ingress (the error message).
If you don't see traffic at port 3, then go to port 2 and repeat the monitoring process.
If you don't see traffic at port 2, then look at the relay node's ($n1p2) network interfaces using "netstat -i" before you enter the URL and after.
My only other suggestion is to talk to your fellow student Mohammad Firas (login toshiba). He seems to have gotten most of the demo working.