For debugging performance issues we found that this script is very
useful. Script collects the stats of vmstat/iostat/top/sar to get
metrics about memory/disk/cpu/network respectively at the configured
interval. It is very important to collect all these metrics at the
same time so that they can be correlated easily, so made this into a
script
fixes bz#1633926
Change-Id: Iea52e55179b21c505953722c8898293fe92cb8de
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
If a syncenv_task starts after syncenv_destroy, the syncenv_task enters
a 600s timeout cond timedwait, and syncenv_destroy must waits it timeout.
Change-Id: I972a2b231e50cbebd3c71707800e58033e40c29d
updates: bz#1626313
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
1. Fix imports
2. Use universal_newlines flag in Popen
3. encode msg used in 'sendto'
Updates: #411
Change-Id: Ieacdc2ba86953ff3a2e9d9b9a97157e8e0e36836
Signed-off-by: Kotresh HR <khiremat@redhat.com>
1. ctypes/syscalls
A) arguments is expected to be encoded
B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str
Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR <khiremat@redhat.com>
CentOS5 too
It's been a couple of years since we stopped doing anything with
RHEL/CentOS 5
Change-Id: I83e78fa84b93fa5baddaf3fab92736d692a9bbca
updates: bz#1193929
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Problem: Current resource cleanup sequence is not
perfect while brick mux is enabled
Solution: 1) Destroying xprt after cleanup all fd associated
with a client
2) Before call fini for brick xlators ensure no stub
should be running on a brick
Change-Id: I86195785e428f57d3ef0da3e4061021fafacd435
fixes: bz#1631357
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
Problem: local could be NULL
Added condition checks to address this issue
Updates: bz#1622665
Change-Id: I7be7dacc5386a77441385240b43f22d85074b69d
Signed-off-by: Sheetal Pamecha <sheetal.pamecha08@gmail.com>
When geo-replication session is running, run
"gluster get-state" command to test.
https://review.gluster.org/#/c/glusterfs/+/20461/
patch fixes glusterd crash, when we run get-state
command with geo-rep session configured.
Adding the test now.
Fixes: bz#1598345
Change-Id: I56283fba2c782f83669923ddfa4af3400255fed6
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
This is critical because we shouldn't be applying the automated
coding standard (clang-format) tool on these files.
This patchset is done by below steps:
* clang-format -i ${filename}
This creates syntax errors. Fix them using below two commands:
* sed -i -e 's/ @/@/g' ${filename}
* sed -i -e 's/,@/, @/g' ${filename}
With this, these files are having minimum changes requried to
compile, and is as close to the coding standard as possible.
* git rename ${filename} ${filename}.in
Updates: bz#1564149
Change-Id: Icf90f7f81d6fa4400be4826e094fdff8e64508d0
Signed-off-by: Amar Tumballi <amarts@redhat.com>
When dht was converted to xlator_api, the variable 'options' was not
changed to dht_options, the same as was done to all the other xlators
that were converted to xlator_api.
Thus the reference to 'options' in dht.c is not resolved until runtime,
and the RTlinker's search path starts with symbols in the executable,
i.e. ganesha.nfsd's 'options'. (Which is obviously not the right one.)
The unused extern references to 'options' (now dht_options) in nufa.c
and switch.c is curious.
Change-Id: Idf4a5d5fbd39aadfa5a4b529bceea65a3cbdf8f3
updates: bz#1193929
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
This patch reduces the execution time of bug-1559004-EMLINK-handling.t
from ~14 minutes to ~90 seconds. To do so, it creates some fake hard
links directly on the brick instead of creating them through the
volume.
Change-Id: I9715ff1a4eba47574c733d4f28e68f42f56a7d3f
updates: bz#1193929
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Till now, glusterd was generating the volfile path for the snapshot
volume's bricks like this.
/snaps/<snap name>/<brick volfile>
But in reality, the path to the brick volfile for a snapshot volume is
/snaps/<snap name>/<snap volume name>/<brick volfile>
The above workaround was used to distinguish between a mount command used
to mount the snapshot volume, and a brick of the snapshot volume, so that
based on what is actually happening, glusterd can return the proper volfile
(client volfile for the former and the brick volfile for the latter). But,
this was causing problems for snapshot restore when brick multiplexing is
enabled. Because, with brick multiplexing, it tries to find the volfile
and sends GETSPEC rpc call to glusterd using the 2nd style of path i.e.
/snaps/<snap name>/<snap volume name>/<brick volfile>
So, when the snapshot brick (which is multiplexed) sends a GETSPEC rpc
request to glusterd for obtaining the brick volume file, glusterd was
returning the client volume file of the snapshot volume instead of the
brick volume file.
Change-Id: I28b2dfa5d9b379fe943db92c2fdfea879a6a594e
fixes: bz#1635050
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
When glusterd receives a SIGTERM while it receives RPC
connect/disconnect/destroy events, the thread might lead to a crash
while accessing rcu_read_lock () as the clean up thread might have
already freed up the resources. This is more observable when glusterd
comes up with upgrade mode = on during upgrade process.
The solution is to ignore these events if glusterd is already in the
middle of cleanup_and_exit ().
Fixes: bz#1635593
Change-Id: I12831d31c2f689d4deb038b83b9421bd5cce26d9
Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
If use this command every some seconds for example 15s to check gluster brick
status, the glusterd will use about 1G memory in a year. free the value of rsp
in gf_cli_status_cbk. glusterd allocate the value of rsp and send it to cli, but
cli do not free the value, that cause glusterd memory leak.
fixes: bz#1635480
Change-Id: I3f19cd0d4b791ae1b35f9664b3a668b1579f1178
Signed-off-by: shujun10086 <shujun.huang@nokia-sbell.com>
Some of the scripts that have a #!/usr/bin/python3 shebang do not have a
main() like function. These scripts will not get executed but only
imported. They do not need the shebang.
A few others are not installed with 'make install', but do have a main()
like function. These scripts are expected to be used by developers for
different tasks (mostly code generation). Marking these scripts
executable to make it easier to identify them.
Change-Id: I73541471deb7e0830766b804786244e73dfe4221
Updates: #411
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Right now, if no option is provided, the default port is assumed,
which is 24007. Ideally, for 'glusterfsd' processes, it is better
to not assume there are any ports given, so it can start listening
on any port which is available.
This helps us to cleanup the dependencies on glusterd from glusterfsd
at the moment. No changes would be done to glusterd code, but making
the right defaults helps to make glusterfsd more independent process
later.
NOTE: This patch is a reduced version of below set of patches:
* https://review.gluster.org/14613/ &
* https://review.gluster.org/14670/ &
* https://review.gluster.org/14671/
Credits: Prasanna Kumar Kalever <pkalever@redhat.com>
updates: bz#1343926
Change-Id: Ib874e10505e7366dc56ba754458252b67052e653
Signed-off-by: Amar Tumballi <amarts@redhat.com>
Handle py2 and py3 compatibility for pickling and unpickling.
Geo-rep pickles and unpickles reading sys.stdin and sys.stdout streams.
py2 and py3 compatibility expects the streams to be opened in binary
mode but the sys.stdout objects are different in python2
and python3
python2:
>>> type(sys.stdout)
<type 'file'>
python3:
>>> type(sys.stdout)
<class '_io.TextIOWrapper'>
So in order to access binary stream, using sys.stdin.buffer in python3
Updates: #411
Change-Id: I1a633ccdddff5baf0cf05a8b493add39ddf75bd7
Signed-off-by: Kotresh HR <khiremat@redhat.com>
python3 expects byte string for os.write. This works
for both py2 and py3. Fixed the same for geo-rep
mount testing code path.
Updates: #411
Change-Id: I2dfedcb0869457707bcca4d2847ef0d52bff1987
Signed-off-by: Kotresh HR <khiremat@redhat.com>
When control flow reaches negative case code block, we added
"goto out" statement without assigning ret to -1.
This patch assigns return value to -1, before going to the lable "out".
Change-Id: I6db651a3c9ca285ade9ee1ca23d0abe6b36fda3a
updates: bz#1193929
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
On systems where /bin/sh is not Bash, running plain mount.glusterfs
gives the unhelpful error "can't shift that many".
The argument parsing can be a little improved. Adding a check for the
number of arguments, minimal two (Gluster ip:/volume, and mountpoint),
but possibly more (-o, -v etc.).
With the additional check, running 'mount.glusterfs -h' now shows the
following messags:
Usage: /sbin/mount.glusterfs <server>:<volume/subdir> <mountpoint> -o<options>
Options:
man 8 mount.glusterfs
To display the version number of the mount helper: /sbin/mount.glusterfs -V
Change-Id: I50e3ade0c6217fab4155f35ad8cb35d99d52e133
Fixes: bz#1564890
Reported-by: Alexander Zimmermann <alexander.zimmermann96@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The char pointer mdc_xattr_str in function mdc_xattr_list_populate
is malloc'd and doing a strcat into a malloc'd region can
overflow content allocated based on prior contents of the
memory region.
Added a NULL terimation to the malloc'd region to prevent
the overflow, and treat it as an empty string.
Change-Id: If0decab669551581230a8ede4c44c319ff04bac9
Updates: bz#1633930
Signed-off-by: ShyamsundarR <srangana@redhat.com>
The jenkins release-new job runs on a CentOS 7 box, which does not
have python3. As a result it runs (autogen.sh and) configure before
producing the dist tar file, converting all the python3 shebangs to
python2 shebangs in the dist tar file.
Then when that tar file is "carried" to, e.g. Fedora koji build
system to build packages, the shebangs are incorrect, despite having
originally been correct in the git repo.
Change-Id: I5154baba3f6d29d3c4823bafc2b57abecbf90e5b
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
In the case glfs_fini is ongoing, some cache xlators like readdir-ahead,
continues to submit requests. Current rpc submit code ignores
connection status and queues these internally generated requests. These
requests then got cleaned up after inode table has been destroyed,
causing crash.
Change-Id: Ife6b17d8592a054f7a7f310c79d07af005087017
updates: bz#1626313
Signed-off-by: Zhang Huan <zhanghuan@open-fs.com>
Problem: After an upgrade from the version where shared-brick-count
option is not present to a version which introduced this option
causes issue at the mount point i.e, size of the volume at mount
point will be reduced by shared-brick-count value times.
Cause: shared-brick-count is equal to the number of bricks that
are sharing the file system. gd_set_shared_brick_count() calculates
the shared-brick-count value based on uuid of the node and fsid of
the brick. https://review.gluster.org/#/c/glusterfs/+/19484 handles
setting of fsid properly during an upgrade path. This patch assumed
that when the code path is reached, brickinfo->uuid is non-null.
But brickinfo->uuid is null for all the bricks, as the uuid is null
https://review.gluster.org/#/c/glusterfs/+/19484 couldn't reached the
code path to set the fsid for bricks. So, we had fsid as 0 for all
bricks, which resulted in gd_set_shared_brick_count() to calculate
shared-brick-count in a wrong way. i.e, the logic written in
gd_set_shared_brick_count() didn't work as expected since fsid is 0.
Solution: Before control reaches the code path written by
https://review.gluster.org/#/c/glusterfs/+/19484,
adding a check for whether brickinfo->uuid is null and
if brickinfo->uuid is having null value, calling
glusterd_resolve_brick will set the brickinfo->uuid to a
proper value. When we have proper uuid, fsid for the bricks
will be set properly and shared-brick-count value will be
caluculated correctly.
Please take a look at the bug https://bugzilla.redhat.com/show_bug.cgi?id=1632889
for complete RCA
Steps followed to test the fix:
1. Created a 2 node cluster, the cluster is running with binary
which doesn't have shared-brick-count option
2. Created a 2x(2+1) volume and started it
3. Mouted the volume, checked size of volume using df
4. Upgrade to a version where shared-brick-count is introduced
(upgraded the nodes one by one i.e, stop the glusterd, upgrade the node
and start the glusterd).
5. after upgrading both the nodes, bumped up the cluster.op-version
6. At mount point, df shows the correct size for volume.
fixes: bz#1632889
Change-Id: Ib9f078aafb15e899a01086eae113270657ea916b
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
Avoid dispatching events to mgmt after freed,
unreference mgmt after the event_dispatch_destroy.
Change-Id: I5b762b37901de70a955661df0aff95bf055ba4ea
updates: bz#1626313
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
In a previous patch (https://review.gluster.org/20769) we've
added the key length to be passed to dict_* funcs, to remove the need
to strlen() it. This patch moves some code to use it.
Please review carefully.
Compile-tested only!
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
Change-Id: If4f425a9827be7c36ccfbb9761006ae824a818c6
The file objects for python3 by default is opened
in binary mode where as in python2 it's opened
as text by default.
The geo-rep code parses the output of Popen assuming
it as text, hence used the 'universal_newlines' flag
which provides backward compatibility for the same.
Change-Id: I371a03b6348af9666164cb2e8b93d47475431ad9
Updates: #411
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Most of the applications are {c|m}time dependant
and very few are atime dependant. So provide noatime
option to not update atime when ctime feature is
enabled.
Also this option has to be enabled with ctime
feature to avoid unnecessary self heal. Since
AFR/EC reads data from single subvolume, atime
is only updated in one subvolume triggering self
heal.
updates: bz#1593538
Change-Id: I085fb33c882296545345f5df194cde7b6cbc337e
Signed-off-by: Kotresh HR <khiremat@redhat.com>
'os.pipe' returns pair of file descriptors
which are non-inheritable by child processes.
But geo-rep uses te inheritable nature of
pipe fds to communicate between parent and
child processes. Hence wrote a compatiable
pipe routine which works well both with python2
and python3 with inheritable nature.
Updates: #411
Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9
Signed-off-by: Kotresh HR <khiremat@redhat.com>
