Design Arno's company

Posted on di 25 september 2018 in designs • Gewijzigd op 28 augustus 2019

Background

I currently work for a few customers as a programmer and for a few others as a systems administrator for their Linux systems.

Current situation (2018) is a physical servers at two different locations. One is in a local data center with a optic fiber connection and one is in a residential area with a DSL connection.

I want to bring the company more in line with permaculture principles. Reduce impact and resource usage of servers. Reduce dependence on internet connections; host data on local machines.

Collect information

9 ways of observing

  • I wonder...
    • how I can use my skills to bring awareness / change
    • if repairing computers is a worthwhile alternative
    • how to reduce the number of computers I use
    • if technology is still interesting enough to keep persuing
    • if I can take most services out of the data center
    • in case of power shortages, do data centers get preferential treatment
  • Observe energy
    • computer have lots of embodied energy
    • electricity to keep computers running
    • Arno's interest is waning
    • Arno's time and brainpower
    • cycling / driving to customers
  • Observe flow
    • provides a (regular) income stream for Arno
    • cycling or driving routes to customers
    • programming can lead to mental flow
    • flow of data from db to webserver to browser to visitor
    • flow of energy to power computers
    • airflow for cooling components
  • Observe communities
    • internet stack (IPv4, IPv6, DNS, HTTP, HTTPS)
    • customers
    • vendors
    • backup server and production servers
    • data center and their peering agreements
    • networks: VPN, home network
    • webserver and database server
  • Observe patterns
    • regular visits to customers
    • people hire me for my expertise
    • proposed solutions (almost) always increase complexity, only very rarely lower it
    • servers tend to get replaced long before they are end-of-life
    • servers / components need maintenance / replacing
  • Observe edge
    • knowledge of
      • Odoo / ERP systems
      • networking computers
      • hardware
    • combine knowing with doing
    • home network / VPN versus wider internet
    • web / mail server is at the edge
  • Observe limits
    • Arno's interest in continuing this path
    • hardware servicing need special components and tools, most I don't have
    • data on servers is physically on other people's property
    • data center is only accessible after appointment
    • breakdown becomes more likely in the future due to energy constraints, lack of replacement parts from breakdowns in suppy chains and diminishing returns on added complexity
    • existing relations, infrastructure and (contractual) obligations
  • Observe from stillness
    • I'm loosing interest
    • I feel aversion in continuing this path
    • there is no life in technology, only intellectual curiosity
  • Observe past & future
    • major source of income
    • held my interest for over 20 years
    • I don't see a bright future for high tech in the long run
      • approaching energy and resource limits
      • specialized, high precision machines required to make (replacement) components
      • without replacement parts, repair will become impossible

Client interview

  • feelings about current arrangements
    • overall layout
      • server layout works
      • server in data center is too big for the amount of work
      • office at home is compact and works
      • backup server in living room is noisy
    • access
      • VPN allows me access to all servers
      • physical access to the server in the data center needs to be scheduled
    • maintenance
      • current layout needs little ongoing maintenance
      • physical server is a liability in case of a hardware failure
  • needs & wants
    • priorities
      • keep ongoing maintenance low
      • keep time commitment low
      • reduce costs
      • reduce energy usage (both at home and in data center)
    • wants
      • my business activities in line with my values
  • values & vision
    • permaculture
    • vegetarian lifestyle
    • simple living
    • frugality
    • avoid the rat race
    • do my own thing
  • life style
    • events
      • visiting customers
      • visiting the data center
      • visiting the accountant
      • making invoices
      • receiving invoices
      • making changes to systems
    • visitors
      • only very rarely will customers visit me at home
    • transport
      • use feet or bicycle
      • rarely use car
    • time on site
      • when working at home: use home office space
      • when working at customer: use assigned desk
    • income
      • this work provides me with a regular income
    • occupation
      • programming
      • systems administration
      • problem solving with computers
      • automating workflows in companies
  • limiting factors
    • physical
      • sitting on a chair behind a computer for a prolonged period of time leads to neck and back pain
    • emotional
      • dislike driving to customers
    • mental
      • I am helping companies to more efficiently ruin our environment; this is a real mindfuck
      • the complexity of maintaining a whole farm of servers / services
    • resources
      • none
  • personal resources
    • vision
      • computers depend on a stable electricity grid
      • computer components depend on a long and complex supply chain that spans the entire world
      • energy from oil is the resource that enables all this
      • free available energy from oil extraction is fast declining
      • the next crisis will be a big one
      • energy supplies may be disrupted
    • network
      • the people I know and trust
      • the businesses I work with and who trust me
      • the physical connection to the internet
    • skills
      • programming PHP, Python and other languages
      • systems administration for Linux
      • explain things clearly
      • getting things done
      • sell projects
  • financial budget
    • current costs (per month):
      • internet access at home: €44
      • server in data center: €83
  • site related
    • neighbors
      • I notice my neighbors more than they notice me
      • data connection with fibre in data center
      • data connection with DSL (copper) at home
    • duration of stay
      • server at home is powered 24/7
      • server at data center is there all year, powered 24/7 and protected by backup power supply (diesel generator)
    • legal
      • insurance for the server in case of problems at the data center
      • data breaches
    • who owns what
      • router on loan from ISP
      • data center equipment from ISP
      • I own all my other equipment
    • issues like theft, vandalism
      • data center is secured, needs an appointment to access
      • home is low key, only advertises with Wifi name
  • time scale
    • changes to be made 2018Q4 - 2019Q1
  • any other questions?
    • none

Evaluate the information

Possible actions

  • find more tenants for the server in the data center
    • when occupancy goes up, better utilization of resources
  • downsize servers
    • only keep small virtual server (VPS) for web, mail, vpn
    • migrate other services to server at home and proxy via VPS when wider access is wanted
    • migrate customer services to a virtual server of their choice
  • limit computing usage
    • power down servers when not in use (on-demand only)
    • shut off internet during the night
  • change activities
    • teach programming
    • teach Linux systems administration
    • host websites
    • create and maintain websites
    • refurbish computers

Apply Permaculture Ethics and Principles

Ethics

  • Earth care
    • Reduce
      • energy consumption
      • number of computers in use
      • send less computers to the landfill
    • Reuse
      • computers
      • components
    • Recycle
      • failed components
    • Repair
      • older computers with salvaged components
  • People care
    • help disadvantaged families
  • Fair share / Future care
    • teach people alternative operating systems and programs like Linux

Attitudinal principles

  • Multiple elements x multiple functions
    • combine services on one machine
    • split services across two machines, backup each on the other
  • Everything gardens
    • simple web/mail server in data center is much cheaper and less resource hungry than a co-located server
    • need to rework agreements with customers and migrate them
    • backup needs be reworked or moved elsewhere if services are moved to home network
  • The problem is the solution
    • small virtual server limited database power
      • convert to static website
      • use proxy to access more powerful machine at home
    • no internet = no access to bookkeeping or other services
      • move data and services to home network
  • Yield is theoretically unlimited
    • with small vitual server in data center
      • resources shared with other customers
      • cost less money (approx. €10 per month)
    • with servers at home
      • can use older computers
      • computer do not need to fit data center space requirements (e.g. does not need to have a fixed width and a limited set of heights)
  • Work with nature
    • database at leaf, webserver at trunk
  • Minimum effort, maximum effect
    • end contract now to create momentum and deadline
    • reuse existing infrastructure
    • I have 1 server running 6 containers. That's seven (virtual and physical) machines that all need to be monitored and kept up to date. Eliminating the physical server removes a lot of complexity.

Holmgren principles

  • Observe and interact
    • electricity use from computing is rising
    • life cycle of servers lengthens (now 5 year depreciation)
    • virtualization / containerization is popular but leads mostly to unnecessary complexity
    • data centers get bigger and more resource hungry
    • download speeds from ISP to home location over DSL are 10x higher than upload speeds
    • network is less congested during the night
  • Catch and store energy
    • salvage and use old computers and components
    • money is a proxy for energy; spend less to conserve energy
  • Obtain a yield
    • spend less money compared to bigger setup
    • administration of a single server is easier
    • teach other people to use Linux
  • Apply self regulation and accept feedback
    • slowly transition to a new situation
    • fix problems as they arise
    • take many small steps
  • Use and value renewable resources and services
    • power computers only during daytime to more likely use intermittent power
  • Produce no waste
    • use secondhand computers
    • limit bandwidth to 80% of max
    • power off computers when not in use and at night
    • consider one computer at home is already a production server
  • Design from patterns to details
    • use functions and elements analysis
    • place data close to where it is needed / processed
  • Integrate rather than segregate
    • combined services on the same physical system
    • backups need to be done in two physically seperate locations to be effective
  • Use small and slow solutions
    • static website
    • external HDD for backup
    • reconsider use of laptops
    • use notification to engage bigger server
      • on request for service at virtual server in datacenter send wake-on-lan to production server on home network; possible implementations: Raspberry Pi always on, connected by VPN or use port forwarding on router
      • requests will be served for a limited time, then server goes back to sleep / powers down
      • downside: extra latency in serving the first request
      • upside: production server is mostly powered down
  • Use and value diversity
    • reuse components from other computers
  • Use edges and value the marginal
    • database at home keeps data close
    • place a backup at a remote location for safe keeping
  • Creatively use and respond to change
    • check providers and prices once a year and change accordingly
    • reconsider this design when circumstances change

Plan for Action

Implement

Tasks per server

Home server (puma)

  • test suspend - works
  • test wake on lan - works, on suspend only
  • test wake on lan from external ip - works, not reliable
  • test suspend after backup - works
  • test wake on alarm for sync and backup - done, works
  • cleanup backup - done

Home server (kitkat)

  • test suspend - works
  • test wake on lan - works, on suspend and poweroff
  • test wake on alarm - works, only on suspend
  • disks too small to hold backups

VPS server (fit1)

  • commission - done
  • configure - done
  • services migration - done
  • test feasibility of running ERP - done

VPS server for customer (syp1)

  • commission - done
  • configure - done
  • services migration - done
  • change DNS - done
  • monitor services - ongoing

Data center server (lion)

  • announce upcoming changes - done
  • migrate own services - done
  • migrate customers away - done
  • remove server from data center - done
  • change DNS - done

Observations

Memory constraints on the VPS lead to swapping when starting Odoo. After the Operating System (OS) finds a new balance, Odoo is usable. Leaving Odoo running triggers the Out of Memory killer (OOM) after a while.

Moving Odoo to a local machine and accessing it via a proxy running on the VPS has proven to work well with only slightly longer response times.

Having many fewer moving parts to worry about and to keep up to date is nice.

Before:

  • 1 server at home
  • 1 server hosting 8 containers in datacenter
  • total: 10 machines to maintain and monitor

After:

  • 1 server at home
  • 2 VPS in datacenter
  • total: 3 machines to maintain and monitor

Maintain

Nothing major changed after migration. Servers still need regular security updates. When new versions of the Debian distribution come out, these will need to be tested and applied.

Currently maintenance consists of:

  • Bi-weekly running (security) updates on all machines.

Evaluate

  • server suspend has no discernible effect on electricity usage
  • Odoo (ERP) is too resource hungry for a small VPS server
    • temporarily installed on VPS, activate by hand (systemctl start odoo) and only when access is needed
    • migrated to home server with a proxy from the VPS location connected via VPN
    • now only accessible when home server is running
  • changeover VPN makes connecting to individual machines more difficult
    • no IPv6 routable network available for VPN network (VPS provider only assigns 32, not a whole /48 subnet)
    • had to assign fixed IPv4 addresses to each client
    • I use .local addresses to address machines by name (using Zero-configuration networking with Avahi)

tweak

  • I found out I can put local addresses (10.x.x.x) in the company DNS. This eliminates the dependency on Avahi.
  • Odoo on the home server works fine as I can activate the server on demand using Wake-on-LAN when I am at home. Most of the time the server is suspended.
  • I have added a suspend after the backup routine has run (with a 5 minute delay). The system automatically wakes up at midnight to start the backup of the day.