Sunday, December 5, 2010

Configuring SHMMAX and SHMALL for Oracle in Linux

Home | Profile 



SHMMAX and SHMALL are two key shared memory parameters that directly impact’s the way by which Oracle creates an SGA. Shared memory is nothing but part of Unix IPC System (Inter Process Communication) maintained by kernel where multiple processes share a single chunk of memory to communicate with each other.

While trying to create an SGA during a database startup, Oracle chooses from one of the 3 memory management models a) one-segment or b) contiguous-multi segment or c) non-contiguous multi segment. Adoption of any of these models is dependent on the size of SGA and values defined for the shared memory parameters in the linux kernel, most importantly SHMMAX.



So what are these parameters - SHMMAX and SHMALL?


SHMMAX is the maximum size of a single shared memory segment set in “bytes”.


silicon:~ #  cat /proc/sys/kernel/shmmax

536870912


SHMALL is the total size of Shared Memory Segments System wide set in “pages”.


silicon:~ #  cat /proc/sys/kernel/shmall

1415577


The key thing to note here is the value of SHMMAX is set in "bytes" but the value of SHMMALL is set in "pages".

What’s the optimal value for SHMALL?


As SHMALL is the total size of Shard Memory Segments System wide, it should always be less than the Physical Memory on the System and should be greater than sum of SGA’s of all the oracle databases on the server. Once this value (sum of SGA’s) hit the limit, i.e. the value of shmall, then any attempt to start a new database (or even an existing database with a resized SGA) will result in an “out of memory” error (below). This is because there won’t be any more shared memory segments that Linux can allocate for SGA.


ORA-27102: out of memory

Linux-x86_64 Error: 28: No space left on device.


So above can happen for two reasons. Either the value of shmall is not set to an optimal value or you have reached the threshold on this server.

Setting the value for SHMALL to optimal is straight forward. All you want to know is how much “Physical Memory” (excluding Cache/Swap) you have on the system and how much of it should be set aside for Linux Kernel and to be dedicated to Oracle Databases.

For e.g. Let say the Physical Memory of a system is 6GB, out of which you want to set aside 1GB for Linux Kernel for OS Operations and dedicate the rest of 5GB to Oracle Databases. Then here’s how you will get the value for SHMALL.

Convert this 5GB to bytes and divide by page size. Remember SHMALL should be set in “pages” not “bytes”.

So here goes the calculation.


Determine Page Size first, can be done in two ways. In my case it’s 4096 and that’s the recommended and default in most cases which you can keep the same. 



silicon:~ # getconf PAGE_SIZE

4096


or

silicon:~ # cat /proc/sys/kernel/shmmni
4096

Convert 5GB into bytes and divide by page size, I used the linux calc to do the math.


silicon:~ # echo "( 5 * 1024 * 1024 * 1024 ) / 4096 " | bc -l

1310720.00000000000000000000


Reset shmall and load it dynamically into kernel


silicon:~ # echo "1310720" > /proc/sys/kernel/shmall
silicon:~ # sysctl –p

Verify if the value has been taken into effect.

silicon:~ # sysctl -a | grep shmall
kernel.shmall = 1310720

Another way to look this up is

silicon:~ # ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096                          /* SHMMNI  */
max seg size (kbytes) = 524288                  /* SHMMAX  */
max total shared memory (kbytes) = 5242880      /* SHMALL  */
min seg size (bytes) = 1


To keep the value effective after every reboot, add the following line to /etc/sysctl.conf


echo “kernel.shmall = 1310720” >> /etc/sysctl.conf

Also verify if sysctl.conf is enabled or will be read during boot.

silicon:~ # chkconfig boot.sysctl
boot.sysctl  on

If returns “off”, means it’s disabled. Turn it on by running

silicon:~ # chkconfig boot.sysctl on
boot.sysctl  on

What’s the optimal value for SHMMAX?

Oracle makes use of one of the 3 memory management models to create the SGA during database startup and it does this in following sequence. First Oracle attempts to use the one-segment model and if this fails, it proceeds with the next one which's the contiguous multi-segment model and if that fails too, it goes with the last option which is the non-contiguous multi-segment model.

So during startup it looks for shmmax parameter and compares it with the initialization parameter *.sga_target. If shmmax > *.sga_target, then oracle goes with one-segment model approach where the entire SGA is created within a single shared memory segment.

But the above attempt (one-segment) fails if SGA size otherwise *.sga_target  > shmmax, then Oracle proceeds with the 2nd option – contiguous multi-segment model. Contiguous allocations, as the name indicates are a set of shared memory segments which are contiguous within the memory and if it can find such a set of segments then entire SGA is created to fit in within this set. 


But if cannot find a set of contiguous allocations then last of the 3 option’s is chosen – non-contiguous multi-segment allocation and in this Oracle has to grab the free memory segments fragmented between used spaces.

So let’s say if you know the max size of SGA of any database on the server stays below 1GB, you can set shmmax to 1 GB. But say if you have SGA sizes for different databases spread between 512MB to 2GB, then set shmmax to 2Gigs and so on.

Like SHMALL, SHMMAX can be defined by one of these methods..

Dynamically reset and reload it to the kernel..


silicon:~ #  echo "536870912" >  /proc/sys/kernel/shmmax

silicon:~ #  sysctl –p           -- Dynamically reload the parameters.

Or use sysctl to reload and reset ..

silicon:~ #  sysctl -w kernel.shmmax=536870912

To permanently set so it’s effective in reboots…

silicon:~ #  echo "kernel.shmmax=536870912" >>  /etc/systctl.conf


Install doc for 11g recommends the value of shmmax to be set to "4GB – 1byte" or half the size of physical memory whichever is lower. I believe “4GB – 1byte” is related to the limitation on the 32 bit (x86) systems where the virtual address space for a user process can only be little less than 4GB. As there’s no such limitation for 64bit (x86_64) bit systems, you can define SGA’s larger than 4 Gig’s. But idea here is to let Oracle use the efficient one-segment model and for this shmmax should stay higher than SGA size of any individual database on the system.




Hope this helps. Regards, Raj

137 comments:

  1. Perfect article! thank you very much for sharing your knowledge!

    ReplyDelete
  2. Great! Thanks a lot for this wonderful article.

    ReplyDelete
  3. Hi - Thanks for the article.

    Then, on a 16GB Linux x86_64 system, what is the max MEMORY_TARGET that can be set? Also, I noticed only if /dev/shm value is increased, the MEMORY_TARGET param can be increased. What would be an ideal setting for a 16 GB machine?

    Thanks

    ReplyDelete
  4. It's Awesome.......

    ReplyDelete
  5. One caveat on your statement "As there’s no such limitation for 64bit (x86_64) bit systems, you can define SGA’s larger than 4 Gig’s. But idea here is to let Oracle use the efficient one-segment model and for this shmmax should stay higher than SGA size of any individual database on the system" :

    11.1.0.7 (11gR1) introduces a bug that is exposed when memory_target and shmmax are higher than 3G and results in ORA-27103: internal error on startup. For details see Doc. ID 743012.1 on MetaLink.

    Rob

    ReplyDelete
    Replies
    1. Thanks Rob, that's good to know. Seems like it's patched from 11.1.0.8; anyways we have few DB's that are on 11.2.0.2 utilizing > 4GB mem with no issues.

      Delete
    2. I think the article is great.
      As for the bug, doesn't seem to be universal
      I have used 11.1.0.6 with higher memory target with no problem ofstartup on centos.
      I think you have to make sure the /dev/shm is also bigger

      nazir

      Delete
    3. Hi, Marvelous article.. simple and easy to under stand.
      Same error is giving for me also. Kindly can u suggest me the value for Shmall and Shmmax .

      Iam getting java pool error and shared pool error so i want to increase the SGA_MAX_SIZE.

      ORA-04031: unable to allocate ORA-04031: unable to allocate 4064 bytes of shared memory ("shared pool")

      ora-04031 unable to allocate 4096 bytes of shared memory ( java pool


      1. Iam having two DB PROD and LUMPROD. SGA_MAX_SIZE is 2gb and 1.5gb. total 3.5gb.

      2. Iam not able to increase sga_max_size for PROD from 2gb to 3gb.

      3. ------ Shared Memory Limits --------
      max number of segments = 4096
      max seg size (kbytes) = 1953124
      max total shared memory (kbytes) = 13118188
      min seg size (bytes) = 1

      cat /etc/sysctl.conf


      # Kernel sysctl configuration file for Red Hat Linux
      #
      # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
      # sysctl.conf(5) for more details.

      # Controls IP packet forwarding
      net.ipv4.ip_forward = 0

      # Controls source route verification
      net.ipv4.conf.default.rp_filter = 1

      # Do not accept source routing
      net.ipv4.conf.default.accept_source_route = 0

      # Controls the System Request debugging functionality of the kernel
      kernel.sysrq = 0

      # Controls whether core dumps will append the PID to the core filename.
      # Useful for debugging multi-threaded applications.
      kernel.core_uses_pid = 1

      kernel.sem = 250 32000 100 150
      kernel.shmmax = 6294967295 (or larger)
      kernel.shmmni = 4096
      kernel.shmall = 3279547
      fs.file-max = 327679
      kernel.msgmni = 2878
      kernel.msgmnb = 360000
      net.ipv4.ip_local_port_range = 1024 65000
      net.core.rmem_default=1048576
      net.core.rmem_max=1048576
      net.core.wmem_default=262144
      net.core.wmem_max=262144
      net.ipv4.tcp_wmem = 262144 262144 262144
      net.ipv4.tcp_rmem = 4194304 4194304 4194304


      cat /proc/sys/kernel/shmall

      3279547

      cat /proc/sys/kernel/shmmax
      1999999999

      M Jain.

      Delete
    4. Can you please paste the output of below commands?

      free -g
      cat /proc/meminfo | grep MemTotal

      Also, I assume you are getting the below while trying to resize the SGA from 2GB to 3GB?

      ORA-27102: out of memory

      Linux-x86_64 Error: 28: No space left on device.

      Delete
  6. Hello Raj,
    We have 2 node RAC using OCFS2 file system.while transferring one file around(800G) suddenly error arise as "No space left on device" even if df -h show available space in that particular LUN.while we gone through the oracle support and different blogs we came to know that was the bug of ocfs2 .support suggest to upgrade the oscf2 version which is not feasible for us right now.now i want to know that how can i calculate that how much space is actually available in the disk to avoid this type of error while transferring huge files in the LUN.Please help.

    Thanks in advance.

    Pankaj

    ReplyDelete
  7. You are getting this while doing a normal file transfer (FTP/SCP ) etc. or an error thrown through Oracle with an "ORA-" error associated with it like below? Thanks.

    "
    ORA-27102: out of memory
    Linux-x86_64 Error: 28: No space left on device.
    "

    ReplyDelete
  8. Awesome detail information, saved my time !!

    ReplyDelete
  9. Hi

    Very Good Explanation. Thank you

    ReplyDelete
  10. Hello,
    thank you so much

    ReplyDelete
  11. Hi Raj,
    On "As SHMALL is the total size of Shard Memory Segments System wide, it should always be less than the Physical Memory on the System and should also be less than sum of SGA’s of all the oracle databases on the server."

    Shouldn't SHMALL be "greater than or equal" to sum of SGA’s of all the oracle databases on the server, instead of "less than"?

    Thanks.

    ReplyDelete
  12. Also on "cat /proc/sys/kernel/shmmni", I believe this is for "max number of shared memory segments segments allocated at a given time" (output you would see as part of ipcs -ml).

    Thanks.

    ReplyDelete
  13. Hi Raj,

    Thanks for this information. This is really helpful.

    -Ganesh.

    ReplyDelete
  14. Hi Raj,

    Thanks for the info! One correction, when making the SHMMAX change permanent there is a typo:
    silicon:~ # echo "kernel.shmmax=536870912" >> /etc/systctl.conf
    it should be
    silicon:~ # echo "kernel.shmmax=536870912" >> /etc/sysctl.conf




    ReplyDelete
  15. max total shared memory (kbytes) = 5242880 /* SHMALL */
    it should be 5242880*1024
    silicon:~ # echo "kernel.shmmax=5368709120" >> /etc/sysctl.conf

    ReplyDelete
  16. Thankx Raj.Great article.Very easz to understand.

    ReplyDelete
  17. Very helpful. Thanks for sharing.

    ReplyDelete
  18. Thanks, nice explanation for shmall.
    But PAGESIZE is always not equal to shmmni. Right?

    ReplyDelete
  19. Shouldn't SHMALL be "greater than or equal" to sum of SGA’s of all the oracle databases on the server, instead of "less than"?

    ReplyDelete
  20. Thanks Raviraj; yes that's right. I just corrected it.

    ReplyDelete
  21. Sometimes doing an echo "" /proc/sys/kernel/shmall might not work. Reason being sysctl -p reads the file /etc/sysctl.conf. So for a permanent fix, I would suggest changing the parameter in: /etc/sysctl.conf and then running sysctl -p

    Cheers,
    Thusjanthan Kubendranathan M.Sc.

    ReplyDelete
  22. I'm a UNIX Admin w/28 yrs experience, and have what looks to be a misconfiguration by my DBA team. On a x86_64 server running Oracle Linux 5.7 with 32GB of physical memory, they are setting:
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    which to me appears to be WAAAAY over the system's limitations. Am I missing something here?

    ReplyDelete
    Replies
    1. Yes that's the wrong configuration. shmall set to 4294967296 which means sum of all SGA's on the server is 16TB (based on 4096Kb pagesize) which can't be possible on 32GB physical memory system. Also shmmax is set to 64GB which is higher than the actual physical memory.

      Delete
  23. awesome article raj... thanks a ton !!

    ReplyDelete
  24. Perl online training|Perl training|call us+919000444287 ...
    www.21cssindia.com/courses/perl-online-training-36.html
    Perl Online Training, Perl Scripting online training by real time Experts from Hyderabad, India. Call 9000444287 for online training demo. Online Perl training ...

    ReplyDelete
  25. Very informative indeed! Thanks for posts like these.

    ReplyDelete
  26. nice information.Thanks for sharing.Page Size Checker is used for checking the total length of a web page.page size checker tool

    ReplyDelete
  27. Collect your Windows Professional Product Keys From Here. Today I Provide you working Windows 8 Professional Product keys or licence key free. If you searching for
    windows 8 pro activation key free ,windows 8 pro activation key free downloads . Source URL

    ReplyDelete
  28. This comment has been removed by the author.

    ReplyDelete

  29. Really very informative and creative contents. This concept is a good way to enhance the knowledge.
    thanks for sharing. please keep it up.
    Linux Training in Gurgaon

    ReplyDelete
  30. This Blog Provides Very Useful and Important Information. I just Want to share this blog with my friends and family members. United States Medical Licensing Examination

    ReplyDelete
  31. The site was so nice, I found out about a lot of great things. I like the way you make your blog posts. Keep up the good work and may you gain success in the long run.

    click here

    Selenium Training in Bangalore|
    Selenium Training in Chennai

    ReplyDelete
  32. Wow.Great post to read.Such a nice and interesting to connect these 2 legand domains.Good job.
    lenovo service centre chennai
    lenovo service center
    lenovo mobile service center near me

    ReplyDelete
  33. Thanks for the post and it is very useful information about a mobile phones. Vivo Service Center in Chennai
    Onepluse Service Center in Chennai

    ReplyDelete
  34. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving.uipath training in bangalore

    ReplyDelete
  35. That was really a great Article.Thanks for sharing information. Continue doing this.

    Real Time Experts Training Center is one of the Best SAP PP Training Institutes in Bangalore with 100% placement support. SAP Production Planning training in Bangalore provided by sap pp certified experts and real-time working professionals with handful years of experience in real time sap pp projects.

    ReplyDelete
  36. Such a great word which you use in your article and article is amazing knowledge. thank you for sharing it.

    Looking for Salesforce CRM Training in Bangalore, learn from eTechno Soft Solutions Salesforce CRM Training on online training and classroom training. Join today!

    ReplyDelete
  37. This is one of the high-quality assets I even have located in pretty a while.
    Once Again Thanks for Sharing this Valuable Information i love this i Can Share this with My Friend Circle.

    click here for more info.

    ReplyDelete
  38. Hi, I do believe this is an excellent website. I stumbledupon it ;) I will come back once again since I bookmarked it. Money and freedom is the best way to change, may you be rich and continue to help others.
    Tech geek

    ReplyDelete
  39. This comment has been removed by the author.

    ReplyDelete
  40. Thanks for sharing this informations.
    hadoop training in coimbatore

    C and C++ training in coimbatore

    embedded training institute in coimbatore

    RPA Course in coimbatore

    Robotic process automation training in coimbatore

    Blue prism training in coimbatore

    Ui Path training in coimbatore

    ReplyDelete
  41. Thank you so much for sharing this messages.
    artificial intelligence training in coimbatore

    DevOps Training institute in coimbatore

    Devops Certification in coimbatore

    Blue prism training in coimbatore

    RPA Course in coimbatore

    C and C++ training in coimbatore

    Robotic process automation training in coimbatore

    ReplyDelete
  42. http://vietnamgsm.vn/members/thanhlymaytinh.159343/

    https://www.techrum.vn/members/thanhlymaytinh.108602/#about

    https://svdhbk.com/members/thanhlymaytinh.29128/

    https://freewebsitetemplates.com/members/thanhlymaytinh/

    ReplyDelete
  43. https://dzone.com/users/4361375/manhinhcu.html

    https://www.flickr.com/people/187466161@N05/

    https://photopeach.com/user/manhinhcu

    ReplyDelete
  44. Hiện nay nhu cầu sử dụng pc ngày càng trở nên phổ biến do nhu cầu học tập và làm việc ở thời đại 4.0 như hiện nay. Tuy nhiên, không phải ai cũng đủ kinh phí để sở hữu một chiếc pc mới, lúc này thì nhiều người lựa chọn phương án mua pc cũ để sử dụng. Tuy nhiên, với thị trường pc khá rầm rộ như hiện nay thì để tìm được địa chỉ bán pc cũ giá rẻ uy tín không phải là vấn đề đơn
    Địa chỉ bán pc cũ giá rẻ uy tín tại Hà Nội

    ReplyDelete
  45. Cpu máy tính là một trong những thiết bị không thể thiếu của một bộ máy tính để bàn. Để có thể lựa chọn được chiếc cpu phù hợp với máy tính của mình thì ngoài những yếu tố như cấu hình, hiệu năng thì vấn đề giá cả hẳn là thông tin mà hầu hết người tiêu dùng băn khoăn khi có dự định tìm mua loại linh kiện máy tính này.

    Báo giá một số cpu máy tính giá tốt tháng 5 tại Ngọc Tuyền

    ReplyDelete
  46. Cpu máy tính là một trong những thiết bị không thể thiếu của một máy tính để bàn trọn bộ. Việc cpu bị tăng nhiệt trong quá trình sử dụng không phải là chuyện hiếm gặp. Tuy nhiên, làm thế nào để giảm nhiệt cho cpu của máy tính thì không phải ai cũng có kinh nghiệm. Nếu bạn cũng đang sử dụng máy tính và chưa tìm ra cho mình giải pháp để giảm nhiệt cho cpu của máy tính trong quá trình sử dụng thì hãy tham khảo thông tin ở bài viết dưới đây nhé.
    Cách làm giảm nhiệt cho cpu máy tính ngày nắng nóng

    ReplyDelete
  47. Cho dù đó là mùa xuân, mùa hè, mùa thu hay mùa xuân, côn trùng có thể là một vấn đề lớn bất cứ lúc nào trong năm. Tùy thuộc vào mùa nào đang đến, bạn có thể phải đối phó với các loại côn trùng gây bệnh khác nhau. Vậy, vào mùa hè, loại côn trùng nào phát triển mạnh nhất. Nếu bạn cũng đang băn khoăn với thông tin này, hãy tham khảo thông tin ở bài viết dưới đây cùng cửa lưới Hoàng Minh nhé
    7 loài côn trùng gây hại phát triển mạnh vào mùa hè

    ReplyDelete
  48. https://www.misterpoll.com/users/594440
    http://www.effecthub.com/user/1775616
    https://www.gta5-mods.com/users/maytinhdebantronbo
    http://groupspaces.com/maytinhdebantronbo/?utm_medium=email&utm_source=notification&utm_campaign=newgroup&utm_term=newgroup

    ReplyDelete
  49. http://noxiousroleplay.nn.pe/member.php?action=profile&uid=19277#tab1
    https://www.geni.com/people/maytinhdebancorei5-ngoctuyen/6000000139681725854#/tab/source
    https://www.geni.com/people/maytinhdebancorei5/6000000139681725854
    https://www.religiousforums.com/members/maytinhdebancorei5.69220/

    ReplyDelete
  50. Cpu máy tính là một trong những thiết bị không thể thiếu của một máy tính để bàn trọn bộ. Việc cpu bị tăng nhiệt trong quá trình sử dụng không phải là chuyện hiếm gặp. Tuy nhiên, làm thế nào để giảm nhiệt cho cpu của máy tính thì không phải ai cũng có kinh nghiệm. Nếu bạn cũng đang sử dụng máy tính và chưa tìm ra cho mình giải pháp để giảm nhiệt cho cpu của máy tính trong quá trình sử dụng thì hãy tham khảo thông tin ở bài viết dưới đây nhé.

    Cách làm giảm nhiệt cho cpu máy tính ngày nắng nóng

    ReplyDelete
  51. https://forum.rigsofrods.org/members/cardmanhinhcuhanoi.13852/#about
    https://www.behance.net/cardmanngoctuy
    https://www.dohtheme.com/community/members/cardmanhinhcuhanoi.18433/
    https://foursquare.com/user/585333088
    https://www.indiegogo.com/individuals/24013628

    ReplyDelete
  52. https://diigo.com/0ih2wj
    https://diigo.com/0ih2wt
    https://diigo.com/0ih2wz
    https://diigo.com/0ih2xe
    https://diigo.com/0ih2xj

    ReplyDelete
  53. https://www.diigo.com/annotated/d40a79e99f815c037239c4891e08de8a
    https://www.diigo.com/annotated/d271201e8b34fae7889fd293f4486afb
    https://www.diigo.com/annotated/1886efc39ba25e0bee1b4cdfb07c480d
    https://www.diigo.com/annotated/efc470b4622009869008bd946858b807

    ReplyDelete
  54. https://leuxonghoimini.blogspot.com/2020/10/cach-chua-au-au-cang-thang-voi-tinh-dau.html
    https://blogduocpham.blogspot.com/2020/10/tuyet-chieu-giam-au-au-cang-thang-bang.html
    https://blogduocpham.blogspot.com/2020/10/tuyet-chieu-giam-au-au-cang-thang-bang.html
    https://tonghopmeovatcuocsong247.blogspot.com/2020/10/meo-hay-giam-au-au-cang-thang-voi-tinh.html
    https://banleuxonghoi.blogspot.com/2020/10/dung-tinh-dau-gung-giam-au-au-cach-nao.html
    https://cungcaptuixonghoi.blogspot.com/2020/10/bat-mi-cach-giam-au-au-cang-thang-voi.html

    ReplyDelete

  55. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

    Data Scientist Course in pune

    ReplyDelete
  56. Really very informative and creative contents. This concept is a good way to enhance the knowledge.
    thanks for sharing. please keep it up.
    DevOps Training in Chennai

    DevOps Course in Chennai

    ReplyDelete
  57. Good Post! it was so good to read and useful to improve my knowledge as an updated one, keep blogging. After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts like this.

    Data Science Training in Gurgaon
    Data Analytics Training in Gurgaon

    ReplyDelete
  58. Thank you so much for sharing this blog with us, it is really amazing valuable and informative.
    Advance Digital Marketing Training in Gurgaon

    ReplyDelete
  59. Hello everyone, We are Children's Furniture – Bao An Kids. Surely when you come to this page, you are looking for a professional children's interior design and construction unit to help you advise and choose for your family and baby to have a perfect and comfortable living space. the safest. Thank you very much for trusting and wanting to accompany Bao An Kids. Phòng ngủ trẻ em

    ReplyDelete
  60. This comment has been removed by the author.

    ReplyDelete
  61. This comment has been removed by the author.

    ReplyDelete
  62. Hello!,I love your writing very so much! percentage we be
    in contact more approximately..
    You can safely browse our links for more information about our services....
    https://erasolutionrbv.com/product/rohypnol-flunitrazepam-2mg/

    ReplyDelete
  63. Very Informative blog thank you for sharing. Keep sharing.

    Best software training institute in Chennai. Make your career development the best by learning software courses.

    android course in chennai
    devops training in chennai
    best devops training in chennai

    ReplyDelete
  64. Get the answer for the query “ How To Get a Job in Infosys as a Fresher? ” with the real-time examples and best interview questions and answers from the best software training institute in Chennai, Infycle Technologies. Get the best software training and placement with the free demo and great offers, by calling +91-7504633633, +91-7502633633.

    ReplyDelete
  65. Стоимость предложений у московских шлюшек определяется почаще всего по медли (за час, ночь). Кроме такого, предусматриваются пожелания клиента - какие непосредственно облики секса он желает получить, чем больше эксклюзивный характер носит обслуживание Элитная индивидуалка проститутока, тем цена девочки дороже. За нестандартные предложения потребуется платить больше, однако симпатичным самцам, каким, непременно, считаешься и ты, девченки могут сделать.

    ReplyDelete
  66. Road cover safe be offer. Appear song character. Bill know with society. Low approach yard contain news agreement front.education

    ReplyDelete