Locals said the incident took place in the afternoon when some people were cutting the hill.
Of the deceased, one was identified as Khoka, 45.
Fire Service Chattogram Station officer Enamul Haque said they have started operation to rescue those workers who trapped under the chunks of mud. Details on missing people would be known after the completion of rescue work, he said.
Deputy Commissioner Abul Bashar Md Fakhruzzaman has visited the spot in the evening.
A seven-member probe body was formed to investigate into the incident, said the DC after visiting the spot.
The committee has been asked to submit its report over the incident within five working days.