seasharp
/
anvil
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

* Updated tools/striker-manage-install-target to now check to see if the striker is a RHEL host. If not, the packages in the RHEL High Availability add-on are merged into the main package list. If it is RHEL though, a search is made for nodes that it can access and once one is found that is a matching RHEL version / arch, and has Internet access, it is used as a proxy to download the packages in the HA add on and then pulls those packages to the local repo.

Browse Source 
* Updated Get->os_type() to work on local and remote hosts.
* Fixed a but in tools/striker-initialize-host where calls to Network->find_matches() where being checked properly for success/failure.

Signed-off-by: Digimer <digimer@alteeve.ca>
main
Digimer 4 years ago
parent 713f77bc78
commit 7e6e345513
8 changed files with 529 additions and 34 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 42
      Anvil/Tools/Get.pm
  2. 2
      Anvil/Tools/Storage.pm
  3. 1
      scancore-agents/scan-apc-ups/scan-apc-ups
  4. 5
      share/words.xml
  5. 8
      tools/striker-initialize-host
  6. 265
      tools/striker-manage-install-target
  7. 232
      tools/test.pl

42
Anvil/Tools/Get.pm
Unescape View File

@ -1271,7 +1271,7 @@ sub md5sum
=head2 os_type
This returns the operating system type and the system architecture as two separate string variables.
This returns the operating system type and the system architecture as two separate string variables. This can be called on the local system, or against a remote system.
# Run on RHEL 8, on a 64-bit system
my ($os_type, $os_arch) = $anvil->Get->os_type();
@ -1281,7 +1281,25 @@ This returns the operating system type and the system architecture as two separa
If either can not be determined, C<< unknown >> will be returned.
This method takes no parameters.
Paramters;
=head3 password (optional)
If C<< target >> is set, this is the password used to log into the remote system as the C<< remote_user >>. If it is not set, an attempt to connect without a password will be made (though this will usually fail).
=head3 port (optional, default 22)
If C<< target >> is set, this is the TCP port number used to connect to the remote machine.
=head3 remote_user (optional, default 'root')
If C<< target >> is set, this is the user account that will be used when connecting to the remote system.
=head3 target (optional)
If set, the os type of the target machine is determined. This must be either an IP address or a resolvable host name.
If not set, the local system's OS type is checked.
=cut
sub os_type
@ -1295,13 +1313,31 @@ sub os_type
my $os_type = "unknown";
my $os_arch = "unknown";
my $password = defined $parameter->{password} ? $parameter->{password} : "";
my $port = defined $parameter->{port} ? $parameter->{port} : 22;
my $remote_user = defined $parameter->{remote_user} ? $parameter->{remote_user} : "root";
my $target = defined $parameter->{target} ? $parameter->{target} : "";
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => $debug, list => {
password => $anvil->Log->is_secure($password),
port => $port,
remote_user => $remote_user,
target => $target,
}});
### NOTE: Examples;
# Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
# Red Hat Enterprise Linux Server release 7.5 (Maipo)
# CentOS Linux release 7.5.1804 (Core)
# Read in the /etc/redhat-release file
my $release = $anvil->Storage->read_file({file => $anvil->data->{path}{data}{'redhat-release'}});
my $release = $anvil->Storage->read_file({
debug => $debug,
file => $anvil->data->{path}{data}{'redhat-release'},
port => $port,
password => $password,
remote_user => $remote_user,
target => $target,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => $debug, list => { release => $release }});
if ($release =~ /Red Hat Enterprise Linux .* (\d+)\./)
{

2
Anvil/Tools/Storage.pm
Unescape View File

@ -2391,7 +2391,7 @@ sub record_md5sums
=head2 rsync
This method copies a file or directory (and its contents) to a remote machine using C<< rsync >> and an C<< expect >> wrapper.
This method copies a file or directory (and its contents) to or from a remote machine using C<< rsync >> and an C<< expect >> wrapper.
This supports the source B<< or >> the destination being remote, so the C<< source >> or C<< destination >> paramter can be in the format C<< <remote_user>@<target>:/file/path >>. If neither parameter is remove, a local C<< rsync >> operation will be performed.

1
scancore-agents/scan-apc-ups/scan-apc-ups
Unescape View File

@ -3001,6 +3001,7 @@ sub find_upses
{
my ($anvil) = @_;
### TODO: On nodes and DR hosts, limit this to the UPSes selected as powering the given machine.
# Search in 'upses' for UPSes using this scan agent. These aren't bound to hosts (or even Anvil!
# systems), so not all may be available.
my $query = "

5
share/words.xml
Unescape View File

@ -1343,6 +1343,11 @@ About to try to download aproximately: [#!variable!packages!#] packages needed t
<key name="message_0181">The scan agent: [#!variable!agent_name!#] check if it's schema was loaded! This is likely a problem with the SQL schema in the file: [#!variable!file!#]. Details are likely available in the: [#!data!path::log::main!#] log file.</key>
<key name="message_0182">The scan agent: [#!variable!agent_name!#] has now successfully loaded! Whatever issue existed with: [#!variable!file!#] has been resolved.</key>
<key name="message_0183">The SQL schema for the scan agent: [#!variable!agent_name!#] has been loaded into the database host: [#!variable!host_name!#].</key>
<key name="message_0184">This Striker is a RHEL host. As such, we'll need to download any updated to packages in the High Availability repositories from entitled nodes. Will search now for a node to use...</key>
<key name="message_0185">The node: [#!variable!node_name!#] is online, has internet access and it is a RHEL machine. Will use it to download HA packages.</key>
<key name="message_0186">No RHEL-based nodes are available. Unable to check for updated packages under the High Availability entitlement.</key>
<key name="message_0187">Downloaded and copied HA packages that started with the letter: [#!variable!letter!#].</key>
<key name="message_0188">Finished downloading HA packages!</key>
<!-- Success messages shown to the user -->
<key name="ok_0001">Saved the mail server information successfully!</key>

8
tools/striker-initialize-host
Unescape View File

@ -262,10 +262,14 @@ sub add_databases
first => $target,
second => $uuid,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { match => $match }});
my $keys = keys %{$match};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
match => $match,
'keys' => $keys,
}});
# Did we find a match?
if ($match)
if ($keys)
{
# Yup!
my $match_found = 0;

265
tools/striker-manage-install-target
Unescape View File

@ -1015,6 +1015,21 @@ sub update_install_source
print $anvil->Words->string({key => "message_0077", variables => { directory => $packages }})."\n";
update_progress($anvil, $progress, "message_0077,!!directory!".$packages."!!");
# If this host is not a RHEL host, add the HA packages to the main packages. If it is RHEL,
# we'll pull the packages from a node.
if ($anvil->data->{host_os}{os_type} ne "rhel8")
{
foreach my $letter (sort {$a cmp $b} keys %{$anvil->data->{ha_packages}})
{
foreach my $package (sort {$a cmp $b} @{$anvil->data->{packages}{$letter}})
{
# Push the package onto the normal array.
push @{$anvil->data->{packages}{$letter}}, $package;
}
}
delete $anvil->data->{ha_packages};
}
foreach my $letter (sort {$a cmp $b} keys %{$anvil->data->{packages}})
{
$progress += 2;
@ -1164,6 +1179,206 @@ sub update_install_source
}
# Progress is '82' leaving this loop
}
# If this is a RHEL host, we'll now look for a node to download HA packages from.
if ($anvil->data->{host_os}{os_type} eq "rhel8")
{
# Try to find a node to download the RPMs on.
update_progress($anvil, ++$progress, "message_0184");
my $use_node_name = "";
my $use_node_ip = "";
my $use_password = "";
my $local_short_host_name = $anvil->Get->short_host_name;
$anvil->Network->load_ips({
debug => 3,
host => $local_short_host_name,
});
my $query = "
SELECT
a.host_uuid,
a.host_name,
b.anvil_password
FROM
hosts a,
anvils b
WHERE
a.host_type = 'node'
AND
(
a.host_uuid = b.anvil_node1_host_uuid
OR
a.host_uuid = b.anvil_node2_host_uuid
)
ORDER BY
a.host_name ASC
;";
my $results = $anvil->Database->query({query => $query, source => $THIS_FILE, line => __LINE__});
my $count = @{$results};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
results => $results,
count => $count,
}});
foreach my $row (@{$results})
{
my $host_uuid = $row->[0];
my $host_name = $row->[1];
my $anvil_password = $row->[2];
my $short_host_name = $host_name;
$short_host_name =~ s/\..*$//;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
host_uuid => $host_uuid,
host_name => $host_name,
anvil_password => $anvil->Log->is_secure($anvil_password),
short_host_name => $short_host_name,
}});
$anvil->Network->load_ips({
debug => 3,
host_uuid => $host_uuid,
host => $short_host_name,
});
my $access = 0;
my ($match) = $anvil->Network->find_matches({
debug => 3,
first => $local_short_host_name,
second => $short_host_name,
});
my $keys = keys %{$match};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { 'keys' => $keys }});
if ($keys)
{
foreach my $interface (sort {$a cmp $b} keys %{$match->{$short_host_name}})
{
my $remote_ip = $match->{$short_host_name}{$interface}{ip};
my $pinged = $anvil->Network->ping({
ping => $remote_ip,
count => 1,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
remote_ip => $remote_ip,
pinged => $pinged,
}});
if ($pinged)
{
my $access = $anvil->Remote->test_access({
target => $remote_ip,
password => $anvil_password,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { access => $access }});
if ($access)
{
my $internet = $anvil->Network->check_internet({
debug => 3,
target => $remote_ip,
password => $anvil_password,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { internet => $internet }});
if ($internet)
{
my ($os_type, $os_arch) = $anvil->Get->os_type({
debug => 3,
target => $remote_ip,
password => $anvil_password,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
os_type => $os_type,
os_arch => $os_arch,
}});
if (($anvil->data->{host_os}{os_type} eq $os_type) && ($os_arch eq $anvil->data->{host_os}{os_arch}))
{
update_progress($anvil, ++$progress, "message_0185,!!node!".$host_name."!!");
$use_node_name = $host_name;
$use_node_ip = $remote_ip;
$use_password = $anvil_password;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
use_node_name => $use_node_name,
use_node_ip => $use_node_ip,
use_password => $anvil->Log->is_secure($use_password),
}});
last;
}
}
}
}
}
}
}
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { use_node_ip => $use_node_ip }});
if ($use_node_ip)
{
foreach my $letter (sort {$a cmp $b} keys %{$anvil->data->{ha_packages}})
{
my $download_path = "/tmp/Packages/".$letter;
my $local_path = "/var/www/html/".$anvil->data->{host_os}{os_type}."/".$anvil->data->{host_os}{os_arch}."/os/Packages/".$letter;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
letter => $letter,
download_path => $download_path,
local_path => $local_path,
}});
# This is the directory we'll download the packages to on the node.
$anvil->Storage->make_directory({
debug => 3,
directory => $download_path,
target => $use_node_ip,
password => $use_password,
mode => "0775",
});
my $packages = "";
my $shell_call = $anvil->data->{path}{exe}{dnf}." download --destdir ".$download_path." ";
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 3, list => { shell_call => $shell_call }});
foreach my $package (sort {$a cmp $b} @{$anvil->data->{ha_packages}{$letter}})
{
# Append the package to the active shell call.
$packages .= $package." ";
}
$packages =~ s/ $//;
$shell_call .= " ".$packages;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { shell_call => $shell_call }});
# None of the HA packages are large os it's not worth trying to monitor the downlaods
# in real time. As such, we'll make a standard remote call.
my ($output, $error, $return_code) = $anvil->Remote->call({
debug => 3,
target => $use_node_ip,
password => $use_password,
shell_call => $shell_call,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
output => $output,
error => $error,
return_code => $return_code,
}});
if (not $return_code)
{
# Success! Copy the files.
my $failed = $anvil->Storage->rsync({
debug => 3,
source => "root\@".$use_node_ip.":".$download_path."/*",
destination => $local_path."/",
password => $use_password,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { failed => $failed }});
if (not $failed)
{
update_progress($anvil, ++$progress, "message_0187,!!letter!".$letter."!!");
}
}
}
update_progress($anvil, ++$progress, "message_0188");
}
else
{
# No nodes found.
update_progress($anvil, ++$progress, "message_0186");
}
}
}
# Create the repodata
@ -1747,17 +1962,6 @@ sub load_packages
"libjpeg-turbo.x86_64",
"libkcapi-hmaccalc.x86_64",
"libkcapi.x86_64",
"libknet1.x86_64",
"libknet1-compress-bzip2-plugin.x86_64",
"libknet1-compress-lz4-plugin.x86_64",
"libknet1-compress-lzma-plugin.x86_64",
"libknet1-compress-lzo2-plugin.x86_64",
"libknet1-compress-plugins-all.x86_64",
"libknet1-compress-zlib-plugin.x86_64",
"libknet1-crypto-nss-plugin.x86_64",
"libknet1-crypto-openssl-plugin.x86_64",
"libknet1-crypto-plugins-all.x86_64",
"libknet1-plugins-all.x86_64",
"libksba.x86_64",
"libldb.x86_64",
"libmaxminddb.x86_64",
@ -1779,7 +1983,6 @@ sub load_packages
"libnl3.x86_64",
"libnma.x86_64",
"libnotify.x86_64",
"libnozzle1.x86_64",
"libnsl2.x86_64",
"liboauth.x86_64",
"libogg.x86_64",
@ -1992,11 +2195,6 @@ sub load_packages
p => [
"p11-kit-trust.x86_64",
"p11-kit.x86_64",
"pacemaker.x86_64",
"pacemaker-cli.x86_64",
"pacemaker-cluster-libs.x86_64",
"pacemaker-libs.x86_64",
"pacemaker-schemas.noarch",
"PackageKit-glib.x86_64",
"PackageKit.x86_64",
"pam.x86_64",
@ -2007,7 +2205,6 @@ sub load_packages
"pciutils.x86_64",
"pcre.x86_64",
"pcre2.x86_64",
"pcs.x86_64",
"perl-aliased.noarch",
"perl-Algorithm-C3.noarch",
"perl-Algorithm-Diff.noarch",
@ -2296,7 +2493,6 @@ sub load_packages
"radvd.x86_64",
"rdma-core.x86_64",
"readline.x86_64",
"resource-agents.x86_64",
"rest.x86_64",
"rootfiles.noarch",
"rpcbind.x86_64",
@ -2446,10 +2642,37 @@ sub load_packages
};
# These packages can't be downloaded on RHEL Striker dashboads as they usually are not entitled to
$anvil->data->{ha_packages} = [
$anvil->data->{ha_packages} = {
c => [
"corosync.x86_64",
"corosynclib.x86_64",
];
],
l => [
"libknet1.x86_64",
"libknet1-compress-bzip2-plugin.x86_64",
"libknet1-compress-lz4-plugin.x86_64",
"libknet1-compress-lzma-plugin.x86_64",
"libknet1-compress-lzo2-plugin.x86_64",
"libknet1-compress-plugins-all.x86_64",
"libknet1-compress-zlib-plugin.x86_64",
"libknet1-crypto-nss-plugin.x86_64",
"libknet1-crypto-openssl-plugin.x86_64",
"libknet1-crypto-plugins-all.x86_64",
"libknet1-plugins-all.x86_64",
"libnozzle1.x86_64",
],
p => [
"pacemaker.x86_64",
"pacemaker-cli.x86_64",
"pacemaker-cluster-libs.x86_64",
"pacemaker-libs.x86_64",
"pacemaker-schemas.noarch",
"pcs.x86_64",
],
r => [
"resource-agents.x86_64",
],
};
my ($os_type, $os_arch) = $anvil->Get->os_type();
if ($os_type eq "rhel8")

232
tools/test.pl
Unescape View File

@ -33,6 +33,232 @@ print "Connecting to the database(s);\n";
$anvil->Database->connect({debug => 3});
$anvil->Log->entry({source => $THIS_FILE, line => __LINE__, 'print' => 1, level => 2, secure => 0, key => "log_0132"});
my $old_date = "2016/02/25";
my $new_date = $anvil->Convert->format_mmddyy_to_yymmdd({date => "2016/02/25"})
print "The old date: [".$old_date."] -> [".$new_date."]\n";
# These packages can't be downloaded on RHEL Striker dashboads as they usually are not entitled to
$anvil->data->{ha_packages} = {
c => [
"corosync.x86_64",
"corosynclib.x86_64",
],
l => [
"libknet1.x86_64",
"libknet1-compress-bzip2-plugin.x86_64",
"libknet1-compress-lz4-plugin.x86_64",
"libknet1-compress-lzma-plugin.x86_64",
"libknet1-compress-lzo2-plugin.x86_64",
"libknet1-compress-plugins-all.x86_64",
"libknet1-compress-zlib-plugin.x86_64",
"libknet1-crypto-nss-plugin.x86_64",
"libknet1-crypto-openssl-plugin.x86_64",
"libknet1-crypto-plugins-all.x86_64",
"libknet1-plugins-all.x86_64",
"libnozzle1.x86_64",
],
p => [
"pacemaker.x86_64",
"pacemaker-cli.x86_64",
"pacemaker-cluster-libs.x86_64",
"pacemaker-libs.x86_64",
"pacemaker-schemas.noarch",
],
r => [
"resource-agents.x86_64",
],
};
my $use_node_name = "";
my $use_node_ip = "";
my $use_password = "";
my ($os_type, $os_arch) = $anvil->Get->os_type();
$anvil->data->{host_os}{os_type} = $os_type;
$anvil->data->{host_os}{os_arch} = $os_arch;
if ($anvil->data->{host_os}{os_type} eq "rhel8")
{
my $local_short_host_name = $anvil->Get->short_host_name;
$anvil->Network->load_ips({
debug => 3,
host => $local_short_host_name,
});
print "My OS is: [".$anvil->data->{host_os}{os_type}."], [".$anvil->data->{host_os}{os_arch}."]\n";
my $query = "
SELECT
a.host_uuid,
a.host_name,
b.anvil_password
FROM
hosts a,
anvils b
WHERE
a.host_type = 'node'
AND
(
a.host_uuid = b.anvil_node1_host_uuid
OR
a.host_uuid = b.anvil_node2_host_uuid
)
ORDER BY
a.host_name ASC
;";
my $results = $anvil->Database->query({query => $query, source => $THIS_FILE, line => __LINE__});
my $count = @{$results};
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
results => $results,
count => $count,
}});
foreach my $row (@{$results})
{
my $host_uuid = $row->[0];
my $host_name = $row->[1];
my $anvil_password = $row->[2];
my $short_host_name = $host_name;
$short_host_name =~ s/\..*$//;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
host_uuid => $host_uuid,
host_name => $host_name,
anvil_password => $anvil->Log->is_secure($anvil_password),
short_host_name => $short_host_name,
}});
$anvil->Network->load_ips({
debug => 3,
host_uuid => $host_uuid,
host => $short_host_name,
});
my $access = 0;
my ($match) = $anvil->Network->find_matches({
debug => 3,
first => $local_short_host_name,
second => $short_host_name,
});
my $keys = keys %{$match};
print "Node: [".$host_name."]\n";
print "- match: [".$match."], keys: [".$keys."]\n";
if ($keys)
{
foreach my $interface (sort {$a cmp $b} keys %{$match->{$short_host_name}})
{
my $remote_ip = $match->{$short_host_name}{$interface}{ip};
print "- Should be able to reach: [".$short_host_name."] at the IP: [".$remote_ip."]\n";
my $pinged = $anvil->Network->ping({
ping => $remote_ip,
count => 1,
});
if ($pinged)
{
print "- The node is pingable, checking access.\n";
my $access = $anvil->Remote->test_access({
target => $remote_ip,
password => $anvil_password,
});
if ($access)
{
print "- Accessed! Testing Internet...\n";
my $internet = $anvil->Network->check_internet({
debug => 3,
target => $remote_ip,
password => $anvil_password,
});
if ($internet)
{
print "- Has Internet access! Checking OS...\n";
my ($os_type, $os_arch) = $anvil->Get->os_type({
debug => 3,
target => $remote_ip,
password => $anvil_password,
});
print "- The host OS is: [".$os_type."], [".$os_arch."]\n";
if (($anvil->data->{host_os}{os_type} eq $os_type) && ($os_arch eq $anvil->data->{host_os}{os_arch}))
{
print "- Found a match!\n";
$use_node_name = $host_name;
$use_node_ip = $remote_ip;
$use_password = $anvil_password;
last;
}
}
}
else
{
print "- Unable to connect.\n";
}
}
else
{
print "- Unable to ping, skipping.\n";
}
}
}
}
if ($use_node_ip)
{
print "Will download RPMs via: [".$use_node_name."] via IP: [".$use_node_ip."]\n";
foreach my $letter (sort {$a cmp $b} keys %{$anvil->data->{ha_packages}})
{
my $download_path = "/tmp/Packages/".$letter;
my $local_path = "/var/www/html/".$anvil->data->{host_os}{os_type}."/".$anvil->data->{host_os}{os_arch}."/os/Packages/".$letter;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
letter => $letter,
download_path => $download_path,
local_path => $local_path,
}});
# This is the directory we'll download the packages to on the node.
$anvil->Storage->make_directory({
debug => 3,
directory => $download_path,
target => $use_node_ip,
password => $use_password,
mode => "0775",
});
my $packages = "";
my $shell_call = $anvil->data->{path}{exe}{dnf}." download --destdir ".$download_path." ";
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 3, list => { shell_call => $shell_call }});
foreach my $package (sort {$a cmp $b} @{$anvil->data->{ha_packages}{$letter}})
{
# Append the package to the active shell call.
$packages .= $package." ";
}
$packages =~ s/ $//;
$shell_call .= " ".$packages;
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { shell_call => $shell_call }});
# None of the HA packages are large os it's not worth trying to monitor the downlaods
# in real time. As such, we'll make a standard remote call.
my ($output, $error, $return_code) = $anvil->Remote->call({
debug => 3,
target => $use_node_ip,
password => $use_password,
shell_call => $shell_call,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => {
output => $output,
error => $error,
return_code => $return_code,
}});
if (not $return_code)
{
# Success! Copy the files.
my $failed = $anvil->Storage->rsync({
debug => 2,
source => "root\@".$use_node_ip.":".$download_path."/*",
destination => $local_path."/",
password => $use_password,
});
$anvil->Log->variables({source => $THIS_FILE, line => __LINE__, level => 2, list => { failed => $failed }});
}
}
}
else
{
print "No nodes are available to try to download HA packages from.\n"
}
}

Write Preview
Loading…
Cancel
Save