Terminal FireFighting – 10 hours

Droplet went down suddenly at 2pm.

  • IP not accessible
  • domain name not accessible

Error : Server refused to connect.

Tried restarting apache. Failed. Resized droplet, because apparently zpanel bandwidth exceeded. It’s such a weird thing to do. Basically it refuses to even serve zpanel. So how are we supposed to resize the bandwidth! i was trying to edit the Httpd-vhosts-conf for hours. Finally managed to bring back zpanel. Then the website itself, without the images ( because of SSL blocking non secure files ). But also, WP admin “forgot password” function refuses to work. “No such username/password”.

After 7 hours of issuing SSH private key to Ben’s guy, editing the httpd, comparing old and new sites etc, finally Nathan said let’s just reboot to old image.

Got to learn more linux terminal command. there’s so many types of environment and the commands are slightly different. The Mac terminal is so powerful, elusive, and i’ve always wanted to learn. but for what ?? Well, this scenario is a pretty big one where i must use terminal.

  • ssh root@IP
  • rm = remove. rm -r foldername
  • cat = copy ?  I used cat ~/.ssh/id_rsa.pub | ssh root@[your.ip.address.here] “cat >> ~/.ssh/authorized_keys” to copy the public key to the server. more about this later.
  • ls = list foldername ( -s, size ) ( “-H” to produce human readable format sizes (like: 1K 101M 2G) (
  • nano = an interesting editor to edit files on the server. F3 to commit, confirm with Y, control+X to exit

  • cd = enter directory
  • go up one level : cd ..
  • mkdir -p = make directory /anotherfolder/foldername
  • /etc/zpanel/configs/ssl  = ssl location
  • /var = stuff location
  • service apache2 restart/status/stop/start
  • If Can’t connect to DB
    service apache2 stop
    service mysql start
    service apache2 start
  • a2enmod ssl = enable ssl mod
  • scp = overwrite
    scp ./X/* root@ip:/etc/zpanel/configs/ssl = copy all file in local folder X to server on that location through ssh
  • cat ~/.ssh/id_rsa.pub  = View own public key
  • [server] cat ~/.ssh/authorized_keys = to view the approved keys
  • [server] nano ~/.ssh/authorized_keys = to append / remove keys
  • enable mod rewrite : a2enmod rewrite , service apache2 restart
  • php upload size : nano /etc/php5/apache2/php.ini , service apache2 restart
  • see size of each folders in the directory : du -sh *
  • “-H” to produce human readable format sizes (like: 1K 101M 2G)
  • sort biggest files in directory : du -Sh | sort -rh | head -n 15
  • remove directory without prompts : rm -rf foldername
  • make a file : touch index.html
  • to copy directory and content to your local : scp -r username@IP:/this/isyour/directory ~/desktop/

For apache vhost ssl “Virtual Host Override” :

SSLEngine on
SSLCertificateFile /etc/zpanel/configs/ssl/x_com_sg.crt
SSLCertificateKeyFile /etc/zpanel/configs/ssl/x_com_sg.key
SSLCertificateChainFile /etc/zpanel/configs/ssl/x_com_sg_bundle.crt

admin-ajax

From digital ocean : Host Key Warning

If you happened to destroy a droplet directly prior to creating the one that you are connecting to, you may see a message like this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...

If this is the case, your new droplet probably has the same IP address as the old, destroyed droplet, but a different host SSH key. This is fine, and you can remove the warning, by deleting the old droplet’s host key from your system, by running this command:

ssh-keygen -R [your.ip.address.here]

In the end, we just rebuild the entire droplet from zpanel-ufw. ( Uncomplicated firewall )
To rebuild > DO > Destroy > Rebuild from
Must shut down first then can rebuild ( Duh )

Took a snapshot of the working server : “21April-working X site with ssl config done.”

Also copied old certs from the old server to local, then copy to the new server. ls directory to check if they are there.

Key learnings :

  1. after installing zpanel, up the bandwidth limit
  2. enable ufw
  3. change all passwords to 16 digits secure ones
  4. monitor bandwidth usage
  5. monitor DO bandwidth usage

Sometimes when mysql is down, it’s due to lack of free space.
Need to go in via console, and remove the logs.

Good linux command website : copy and paste, rename, mkdir
http://linuxcommand.org/lts0050.php

IF apache2 fails after SSL key / crt upload, goes to /etc/zpanel/configs/apache and check that the vhost file is correct.

Check disk usage : df -h