Commit Graph

340 Commits

Author SHA1 Message Date
Digimer
daca6c887b * This contains a fairly major change to how time stamps are handled. All INSERT and UPDATE calls now generate a new timestamp via Database->refresh_timestamp, instead of using 'sys::database::timestamp'. This was done in responce to finding a bug where tables in a database differed in both counts of public and private schemas (ip_addresses table, specifically) that failed to resync because the timestamps were re-used too often.
* WIP - Continuing work on the new anvil-manage-server tool.
* Updated Database->get_anvils() to load information on the files available on each Anvil! system.
* Updated Database->insert_or_update_network_interfaces() to no longer take the 'timestamp' parameter.
* Removed all logging from Database->refresh_timestamp() to speed it up, given how often it will be called now.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-06-08 15:23:15 -04:00
Digimer
930b0b6729
Merge branch 'master' into anvil-tools-dev 2021-06-06 00:05:15 -04:00
Digimer
e15c1651ed * Fixed a bug with deleting bad keys where jobs to delete keys on non-dashboard machine wasn't being assigned to the proper target machine.
* Fixed a bug with anvil-manage-keys where a state_uuid entry recorded on one database may not be read from a machine reading from another database.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-06-05 19:07:25 -04:00
Tsu-ba-me
d72d037a01 fix(cgi-bin): correct reserved memory constant to 8GiB 2021-06-04 19:41:09 -04:00
Tsu-ba-me
5a22512665 fix(cgi-bin): add login cookie guard to set_power 2021-06-04 19:18:28 -04:00
Tsu-ba-me
4fd9374722 fix(cgi-bin): add login cookie guard to set_membership 2021-06-04 19:17:49 -04:00
Tsu-ba-me
097900a2e9 fix(cgi-bin): add login cookie guard to get_status 2021-06-04 19:16:51 -04:00
Tsu-ba-me
0eb9c9ff82 fix(cgi-bin): add login cookie guard to get_shared_storage 2021-06-04 19:16:04 -04:00
Tsu-ba-me
60f57c5d35 fix(cgi-bin): add login cookie guard to get_servers 2021-06-04 19:15:13 -04:00
Tsu-ba-me
4b1905b2f5 fix(cgi-bin): add login cookie guard to get_replicated_storage 2021-06-04 19:14:33 -04:00
Tsu-ba-me
9c79d0d07b fix(cgi-bin): add login cookie guard to get_networks 2021-06-04 19:13:25 -04:00
Tsu-ba-me
103fd121e6 fix(cgi-bin): add login cookie guard to get_memory 2021-06-04 19:07:16 -04:00
Tsu-ba-me
97cbe279fc fix(cgi-bin): add login cookie guard to get_cpu 2021-06-04 19:00:34 -04:00
Tsu-ba-me
9fda3af2ce fix(cgi-bin): move error string key to 0307 2021-06-04 19:00:34 -04:00
Tsu-ba-me
4a3c364724 fix(cgi-bin): check login cookie in get_anvils 2021-06-04 19:00:34 -04:00
Tsu-ba-me
c3ac652043 build(cgi-bin): exclude get_anvil_status endpoint 2021-06-04 12:11:48 -04:00
Tsu-ba-me
0eefcdc06e build(cgi-bin): include get_networks endpoint 2021-06-04 12:11:48 -04:00
Tsu-ba-me
eb543e7723 build(cgi-bin): remove unused cold_stop endpoint 2021-06-04 12:11:48 -04:00
Tsu-ba-me
215a96257b fix(cgi-bin): reset log level in get_shared_storage 2021-06-03 11:26:32 -04:00
Tsu-ba-me
4fa1cb7f31 fix(cgi-bin): include storage group UUID in response 2021-06-03 11:26:32 -04:00
Tsu-ba-me
4f88b1f536 fix(cgi-bin): ensure sizes in get_shared_storage's response are JSON numbers 2021-06-03 11:26:32 -04:00
Tsu-ba-me
507588cfc3 fix(cgi-bin): add logic to fetch basic info on storage groups 2021-06-03 11:26:32 -04:00
Tsu-ba-me
b1632c2dc3 fix(cgi-bin): make get_shared_storage executable 2021-06-03 11:26:32 -04:00
Tsu-ba-me
22caf55ca0 fix(cgi-bin): accept query param and switch in get_shared_storage 2021-06-03 11:26:32 -04:00
Tsu-ba-me
0008aec694 chore(cgi-bin): add boilerplate to new get_shared_storage 2021-06-03 11:26:32 -04:00
Tsu-ba-me
8789a3b891 chore(cgi-bin): rename existing get_shared_storage->get_filesystems 2021-06-03 11:26:32 -04:00
Tsu-ba-me
c1a6b1c998 fix(cgi-bin): reset log level in get_networks 2021-06-02 18:33:36 -04:00
Tsu-ba-me
e1c5036fb4 fix(cgi-bin): add switch support to get_networks 2021-06-02 18:33:36 -04:00
Tsu-ba-me
4fa8073efe fix(cgi-bin): add short host name to get_status response 2021-06-02 18:33:36 -04:00
Tsu-ba-me
4fe11050e8 fix(cgi-bin): rename nodes->hosts in get_networks 2021-06-02 18:33:36 -04:00
Tsu-ba-me
2223af6d6c fix(cgi-bin): rename nodes->hosts in get_status 2021-06-02 18:33:36 -04:00
Tsu-ba-me
457db203bd fix(cgi-bin): rename nodes->hosts in get_memory 2021-06-02 18:33:36 -04:00
Tsu-ba-me
2530d2b30d fix(cgi-bin): rename nodes->hosts in get_shared_storage 2021-06-02 18:33:36 -04:00
Tsu-ba-me
8245b095fb fix(cgi-bin): make get_networks executable 2021-06-02 18:33:36 -04:00
Tsu-ba-me
2f2584b185 fix(cgi-bin): use short host name in get_anvils response 2021-06-02 18:33:36 -04:00
Tsu-ba-me
bf055759ee fix(cgi-bin): compute link state in get_networks 2021-06-02 18:33:36 -04:00
Tsu-ba-me
a8f160af48 fix(cgi-bin): add logic to build get_networks response 2021-06-02 18:33:36 -04:00
Tsu-ba-me
b9f9d07362 fix(cgi-bin): handle get_networks input 2021-06-02 18:33:36 -04:00
Tsu-ba-me
d2f34ea5a9 docs(cgi-bin): revise set_power description 2021-06-02 18:33:36 -04:00
Tsu-ba-me
ffe2a7f59e feat(cgi-bin): add boilerplate for get_networks 2021-06-02 18:33:36 -04:00
Tsu-ba-me
7058a967ed fix(cgi-bin): add temporary constant reserved in get_memory response 2021-06-02 18:33:36 -04:00
Tsu-ba-me
10269bf74b fix(cgi-bin): remove server_host_index from get_servers response 2021-06-02 18:33:36 -04:00
Tsu-ba-me
adbf7b790e fix(cgi-bin): rename node->host in response of get_anvils 2021-06-02 18:33:36 -04:00
Tsu-ba-me
8d1d49e807 fix(cgi-bin): add missing response info to get_anvils 2021-06-02 18:33:36 -04:00
Tsu-ba-me
c749441e3b fix(cgi-bin): transform maintenance_mode to JSON booleans in get_status 2021-06-02 18:33:36 -04:00
Tsu-ba-me
e1c3064303 fix(cgi-bin): transform all memory sizes to JSON numbers 2021-06-02 18:33:36 -04:00
Tsu-ba-me
552ed66829 fix(cgi-bin): transform CPU cores and threads to JSON numbers 2021-06-02 18:33:36 -04:00
Tsu-ba-me
78f50e424c fix(cgi-bin): transform FS total size to int in get_shared_storage 2021-06-02 18:33:36 -04:00
Tsu-ba-me
8bcff7a038 fix(cgi-bin): transform boolean int to JSON boolean in get_shared_storage 2021-06-02 18:33:36 -04:00
Tsu-ba-me
5b7367a579 fix(cgi-bin): add file_systems response property to get_shared_storage 2021-06-02 18:33:36 -04:00
Tsu-ba-me
028d547d7c fix(cgi-bin): reset log level in set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
7ffb84f9a3 fix(cgi-bin): set join and leave cluster jobs to run on target host 2021-06-02 15:27:31 -04:00
Tsu-ba-me
52a524564f fix(cgi-bin): try to use switches when failed to parse request body in get_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
7bf604b3e2 fix(cgi-bin): add switch for is_member in set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
50c5098230 fix(cgi-bin): release job dispatching logic in set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
1cc0a9b087 fix(cgi-bin): add switch support to set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
419ec52d2b fix(cgi-bin): add job title and description to set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
d6c1013976 fix(cgi-bin): add join/leave cluster logic to set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
314e8db757 fix(cgi-bin): get host's cluster membership status in set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
da6e3500a3 fix(cgi-bin): add request body parsing to set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
c7fe5aa1e0 fix(cgi-bin): add boilerplate to set_membership 2021-06-02 15:27:31 -04:00
Tsu-ba-me
593921fa04 fix(cgi-bin): reset log level in set_power 2021-06-01 14:15:44 -04:00
Tsu-ba-me
5c7d9d9e82 fix(cgi-bin): stop servers when shutting down the whole anvil 2021-06-01 14:13:29 -04:00
Tsu-ba-me
9a483b5d55 fix(cgi-bin): set shutdown job to run on target node 2021-06-01 14:13:29 -04:00
Tsu-ba-me
0020103b77 fix(cgi-bin): try to use switches when failed to parse request body 2021-06-01 14:13:29 -04:00
Tsu-ba-me
ba93ae38d8 fix(cgi-bin): add switch for is_on in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
7ef54f08a4 style(cgi-bin): remove excess indent in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
d1149d5978 fix(cgi-bin): release job dispatching logic in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
5a3daa3528 fix(cgi-bin): add switch support to set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
e80ebdb8b4 fix(cgi-bin): correct host_uuid existence check in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
f966ed12fb style(cgi-bin): format set_power to keep consistency 2021-06-01 14:13:29 -04:00
Tsu-ba-me
fae1c72986 docs(cgi-bin): rename hash->response_body in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
540129bb8f fix(cgi-bin): remove outdated logic 2021-06-01 14:13:29 -04:00
Tsu-ba-me
7c1e6e7c87 fix(cgi-bin): add proper error messages when given invalid UUID(s) 2021-06-01 14:13:29 -04:00
Tsu-ba-me
1ec32bfbaf fix(cgi-bin): filled in logic to power on/off target host(s) 2021-06-01 14:13:29 -04:00
Tsu-ba-me
12113e1dda fix(cgi-bin): parse set_power request body to get parameters 2021-06-01 14:13:29 -04:00
Tsu-ba-me
3ae54037ea fix(cgi-bin): improve verbosity in set_power 2021-06-01 14:13:29 -04:00
Tsu-ba-me
a8571863be fix(cgi-bin): add subs to handle set anvil/host power 2021-06-01 14:13:29 -04:00
Digimer
41cd1e0319 * Several bugs fixed and enhancements;
* DRBD is now configured to a ping-timeout of 3 seconds.
* Created Log->switches() that returnes the command line switches used by Anvil! tool command line calls based on the active log levels / secure logging. Appended this to all invocations of our tools.
* Updated Database->resync_databases() to now only skip 'jobs' and 'variables' tables with less than 10 record differences. All other differences will trigger a resync.
* Created System->_check_anvil_conf() that, as you might guess, checks in anvil.conf exists and created it (using defaults), if not. It also checks to see if the 'admin' group and user exists and creates them, if not.
* Updated anvil-daemon to check anvil.conf on start up and in each loop. Created the function check_journald() that checks (and sets, if needed) that journald logging is persistent.
* Made striker-manage-peers to check_if_configured on the Database->connect() when updating anvil.conf and the target UUID is the local machine. Also created a loop to make the reconnection a lot more robust.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-05-24 00:09:32 -04:00
Digimer
fc0954d0c8 * Started work on, but not at all finished, anvil-manage-server which will allow manipulation of a server's resources.
* Changed the alteeve repo RPM to the new cimmunity/enterprise repo
* Fixed a bug where 'fence_data::updated' was causing the fences web page to break.
* Fixed a bug in Database->insert_or_update_network_interfaces() where certain interfaces were being repeatedly added to the database.
* Fixed a bug in Database->_find_behind_databases() was marking DBs as behind even though they had less than 10 columns off.
* Fixed a bug in Get->host_name() where, if the host name was changed on disk but the environment variable was still the old name, it would cause the hostname to waffle back and forth and cause constant updated to /etc/hosts.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-05-20 00:16:09 -04:00
Digimer
4a87ee71db * This commit started with work on webui endpoint set_power, but then switched to scancore debugging and I neglected to switch branches.
* Created Cluster->check_stonith_config() that checks and, if needed, reconfigures a cluster's fencing (stonith) config.
* Updated scan-cluster to call Cluster->check_stonith_config() at the end of each call.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-05-02 21:40:48 -04:00
Digimer
a480357049 * Fixed a bug in Cluster->assemble_storage_groups() where, if a group is created during an anvil-provision-server run, the group would get created multiple times.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-15 18:51:10 -04:00
Digimer
798518ba5e * While working on the boot/shutdown server tools, ran into and fixed a bug where files uploaded before an Anvil! was added could not have those files sync'ed. This was fixed though the new Database->check_file_locations() method.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-14 22:56:18 -04:00
Digimer
942e0f66bf * Finished the 'get_X' enpoints so far defined. Added get_servers and completed get_status
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-11 16:26:42 -04:00
Digimer
fb0836f912 * THe get_cpu endpoint was completed.
* The get_mmeory endpoint was completed.
* The get_replicated_storage endpoint was completed, though it requires testing and likely has issues.

To prepare for the get_status endpoint work, I needed to update ScanCore and modules to track the host_status. This commit contains the work needed for this.
* Updated ScanCore->post_scan_analysis_striker() to use configured fence devices (except PDUs) to check if a target host is off or on, in there is no host_ipmi interface. In all cases, if a machine can be confirmed on or off, the host_status is now updated.
* To support the above fence based power checks, updated scan-cluster to store the on-disk CIB in the new scan_cluster -> scan_cluster_cib colume.
* Updated ScanCore->parse_cib() to map stonith primitive IDs to fence agents. Updated ->parse_crm_mon() to not call if the executable doesn't exist to avoid unhelpful error messages in the logs when called from a Striker.
* Update DRBD->gather_data() to get the size data from /sys/block/drbd<minor>/size' x '/sys/block/drbd<minor>/queue/logical_block_size so it works when a device is Secondary (and can't be promoted).
* Updated Database->get_hosts_info() to record the short host name as well as the stored host name. Created ->update_host_status() as a wrapper to ->insert_or_update_hosts() that only updates the host status.
* Updated anvil-join-anvil to disabled ksm and ksmtuned daemons.
* Updated scancore and anvil-daemon to set the host_status to 'online' on startup.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-09 20:51:29 -04:00
Digimer
c2fe3a2f0a * Finished (initial) get_shared_storage.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-02 22:22:07 -04:00
Digimer
fa3c861a97 * Started work again on get_shared_storage
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-04-02 18:31:54 -04:00
Digimer
f3bbec53ef * Fixed cgi-bin/Makefile typo
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-03-22 17:04:39 -04:00
Digimer
5536e8ff47 * Updated Cluster->assemble_storage_groups() and Cluster->anvil_name_from_uuid() and ->available_resources() to try to detect the anvil_uuid if not passed in.
* Updated Database->insert_or_update_storage_group_members() to use the host_uuid when trying to find existing members.
* Added the skeleton of a bunch of new json endpoints for the new UI features.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-03-16 02:40:50 -04:00
Digimer
0fb191c00f * Made more progress on tools/striker-auto-initialize-all, now to the point where it loads the variables needed to initialize Striker dashboard.
* Cleaned up / added some logging in various locations.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-03-02 01:18:18 -05:00
Digimer
e8efbab343 * The work on PXE / UEFI support is broken, and will be set aside for the time being. The commit here is working to getting things fixed, but it's taking too much time away from more pressing issues.
* This commit includes two unrelated test files for UI work, cgi-bin/get_anvil_status and cgi-bin/get_anvils.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-02-16 17:32:43 -05:00
Digimer
bfe5fe2281 * This adds two new files; cgi-bin/get_anvils and cgi-bin/get_anvil_status, that will form the foundation of the new interactive web UI.
* Added the new 'json_headers' template to shared.html
* Updated Database->get_anvils() to store 'anvils::anvil_uuid::<anvil_uuid>::query_time' to aid in JSON clients to know the data is coming from the server. Also fixed a bug where the normal 'modified_date' column wasn't being loaded from the database properly.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-02-09 22:45:40 -05:00
Fabio M. Di Nitto
8f9892650b [build] first pass at adding a build system to integrate with CI
Signed-off-by: Fabio M. Di Nitto <fabbione@fabbione.net>
2021-01-30 20:16:30 +01:00
Digimer
68ea6da1d3 * Finished the web interface components of the Anvil! File Manager! Files can be purged, sync'ed or removed from specific Anvil! systems, renamed and their file types changed (and setting/removing the executable bits) as needed.
* Fixed a bug in Database->insert_or_update_jobs() where the 'job_host_uuid' being set to 'all' only translated to a job for the running host.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-01-06 01:31:09 -05:00
Digimer
6da2b3b17b * Got more work done on file management. A file name is now clickable and that loads a menu to rename, change the file type, purge (delete from everywhere) and select which Anvil! systems the file belongs on. Got the code done to purge a file, but it's not tested yet.
Signed-off-by: Digimer <digimer@alteeve.ca>
2021-01-04 02:05:05 -05:00
Digimer
ea84ba68eb * Fixed a bug in tools/anvil-update-states that was causing deleted interfaces to update the network_interfaces every pass, growing the DB excessively.
* Cleaned up the file manager;
** Got the jquery file uploader JS to be sane and altered it to be more useful.
** Got the list of existing files to be displayed (links clickable but not working yet).

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-01-03 01:14:04 -05:00
Digimer
7d3c4371c7 * Renamed tools/striker-sync-shared to tools/anvil-sync-shared, as it's now designed to run on all machines. Got it to the point where it can be run on Anvil! members to pull down freshly uploaded files. It does so, when two or more strikers are available with the target file, load balancing such that one node downloads from one striker while another node downloads from the other striker. If there is three nodes, and if there is a DR host, the DR host will download from the third striker. If there are 1 or 2 strikers, the DR host will wait to download after both nodes have finished downloading.
* Cleaned up upload.pl now that it isn't responsible for loading the file details into the database. It only sets a job for the local Striker to process the file and move it into /mnt/shared/files, copy it to peer dashboards, then load jobs for Anvil! members to sync the new file.
* Created Database->get_files() and ->get_file_locations() to load the respective data.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-01-02 02:23:33 -05:00
Digimer
1a36f37065 * Got file uploads working!
* Got tools/striker-sync-shared to pick up 'upload::move_incoming' jobs, move the uploaded file to /mnt/shared/files/, copies it to peer dashboards, adds it to the 'files' table and adds it to 'file_locations'.
* Reworked the 'file_locations' table to now map files to Anvil! systems, not hosts. It simply tracks if a given file should be on Anvil! members or not. Later, striker-sync-shared on the Anvil! members will pull the file down.
* Updated Storage->get_file_stats() to record the file's mimetype.
* Fixed up a few issues in cgi-bin/upload.pl.

Signed-off-by: Digimer <digimer@alteeve.ca>
2021-01-01 04:09:17 -05:00
Digimer
713f77bc78 * Finally finished scan-apc-ups! Proved way harder than anticipated... (over a solid week of work!) In M3, this agent is no longer host-bound, and the UPSes to scan based on entries in 'upses' using this scan agent.
* Fixed a bug in Database->insert_or_update_power() where the check to see if 'power_ups_uuid' was passed in was reversed. Also fixed a bug where the convertion of the value to TRUE/FALSE for the old value wasn't being set correctly.
* Updated Server->get_definition() to only translate the host name to a uuid if the host uuid wasn't passed in. Added a sanity check on the UUID as well.
* Cleaned up how existing UPSes are displayed in Striker when managing UPSes. Also renamed the form's scan agents to match the real agent names.
* Fixed alert sorting in scan-apc-pdu.

Signed-off-by: Digimer <digimer@alteeve.ca>
2020-11-12 00:35:51 -05:00
Digimer
0f7267eae1 * Moved the '_host_name', '_short_host_name', and '_domain_name' private methods in Tools.pm over to Get.pm (removing the leading '_' in the method names).
* Created 'Cluster->which_node' that returns 'node1' or 'node2' to indicate which node a host is.
* Continued working on scan_cluster; decided to make it not host-dependent.

Signed-off-by: Digimer <digimer@alteeve.ca>
2020-09-20 00:27:36 -04:00